Clone via ssh from Windows drive D: to Linux?
(1) By ZanyScum (zany) on 2019-09-24 15:27:31 [link] [source]
What I have:
- I have fossil on a Windows 10 system.
- I have on this Windows system a ssh server running and able to be connected to via ssh keys.
- I have a Linux system which can ssh into the Windows system.
- I can connect using the config file, so "ssh pcname" is working
What I want:
- I'd like to use the fossil clone command from Linux on a repository which resides on D: on a Windows system.
My tries so far:
- fossil clone ssh://pcname/D:\_ssh\repositories\test.repo test.repo
- fossil clone ssh://pcname/D:/_ssh/repositories/test.repo test.repo
- fossil clone ssh://pcname//D:/_ssh/repositories/test.repo test.repo
What I get: server did not reply Clone done, sent: 280 received: 81 ip: pcname server returned an error - clone aborted
How do I do it the right way?
(2) By Richard Hipp (drh) on 2019-09-24 15:38:54 in reply to 1 [link] [source]
Seems like the second approach is correct, and it seems to have worked, did it not? What makes you think it didn't work?
The way clone (or sync) via SSH works is that it uses ssh to run the following command on the remote system:
fossil test-http REPONAME
Where REPONAME is the pathname of the ssh: URI. You want
D:/_ssh/repositories/test.repo
and so I think you want the second
URI. The input and output to this command is the ordinary
HTTP traffic that is normally sent over a network socket, though
in this case it travels over the SSH connection instead.
The ssh command expects fossil to be on %PATH%. If it is not, you will need to use a query parameter to specify the Fossil binary:
ssh://pcname/D:/_ssh/repositories/test.repo?fossil=c:/what/ever/fossil.exe
But I don't think you need to do this because apparently your second case worked, at least to judge from the output you got back.
(3.1) By Warren Young (wyetr) on 2019-09-24 17:20:11 edited from 3.0 in reply to 1 [source]
On Windows, Fossil calls "plink -ssh -T
" to set up SSH tunnels by default, which relies on either the plink.exe
from PuTTY or the one from SSH Communications. If you don't want to install plink.exe
from either source, but instead want to use OpenSSH — perhaps the version recently added to Windows — you have to override this with the ssh-command
setting:
c:\> fossil set ssh-command "ssh -e none -T"
ssh.exe
will need to be in the %PATH%
, of course.
You might want to make this setting global, so it affects all repos on that system.
(4) By ZanyScum (zany) on 2019-09-24 18:13:16 in reply to 2 [link] [source]
You directed me in the right direction. After connecting via ssh fossil was not in the path.
The following seems to do ... something:
fossil clone ssh://pcname/D:/_ssh/repositories/test.repo?fossil=C:/myname/bin/win/32/fossil.exe test2.repo
It creates test2.repo with a size of 217088 bytes (versus the original 3MB) then it ... don't know... stops? It keeps running, I can enter text and see it and press return, but it is is like the program is still running or waiting for a special input.
Or if the command above is still missing something.
(5) By ZanyScum (zany) on 2019-09-24 18:32:08 in reply to 3.1 [link] [source]
I am sorry, I am not sure what you are talking about.
On Windows I am calling a ssh server. I do not work with putty, plink or any ssh client at all.
(6) By ZanyScum (zany) on 2019-09-24 18:41:52 in reply to 4 [link] [source]
I just used the -v option:
fossil clone -v ssh://desktop/D:/_ssh/repositories/test.repo?fossil=C:/andreas/bin/win/32/fossil.exe test2.repo
Bytes Cards Artifacts Deltas
waiting for server...
(7) By Richard Hipp (drh) on 2019-09-24 18:58:59 in reply to 4 [link] [source]
I don't know how to configure an ssh server on a Windows machine. That is not something I've ever done before, so cloning off of a Windows machine using ssh is not something I've ever tested. I've done it the other way around - Windows client and unix server - but I've never tried to clone onto a unix client from a Windows server.
Perhaps there is some \r\n translation issues messing up the protocol. The messages that go back and forth from client to server contain byte-counts, and so if extra \r characters get inserted or removed by SSH, then things will not go well. We've worked through all those issues for a Windows client hitting a unix server, but not the other way around.
Or it could be something else. I don't have any way of testing.
Have you considered just running "fossil server" on the Windows machine, then cloning or syncing using ordinary http:?
(8) By ZanyScum (zany) on 2019-09-24 19:38:19 in reply to 7 [link] [source]
Thank you for trying to help.
My situation is the following: I am using privately svn since 15 years or more. Of course not many more ;-)
Now I am trying to go away from 2x desktop + 1x netbook + 1x server (24/7). I want to just remove my Jenkins/Nagios/Subversion/Reddit server and just have a repository on my desktop and another on on my Netbook, perhaps later on a laptop too. So... no server of any kind.
I hate git. I have used it for ca. 3 years on a daily basis and I have learned to really hate it.
So after some research I am currently thinking of using either fossil or mercurial. But copy-one-file-and-your-installation-is-done is such an unbeatable argument, I just HAD to try it.
Now the reason for my wall of text: I will perhaps have to sync different systems via WLan and I will not sync my personal data over WLan and http. Even https would be something I'd like to avoid. I just trust ssh more. No valid reason. It is just that way.
That is the reason I do not want to have fossil running as http server.
A possibility I am not so happy about even if it were possible, but I'd like to know if it IS possible: Can I temporarily start my normal no-server fossil as http server, connect via ssh and get the data from localhost:8080 or whatever port I'd use?
Http over ssh? If it is possible, how I start the http service (Windows and Linux please) and how would the connection line look like?
(9) By ZanyScum (zany) on 2019-09-24 19:40:11 in reply to 8 [link] [source]
Sorry, I am stupid, that is what ssh tunnels are for. I'll try it perhaps the next days.
(10) By Richard Hipp (drh) on 2019-09-24 20:37:31 in reply to 8 [link] [source]
Other options to consider:
Put Fossil on a raspberry pi sitting somewhere in the corner, and use ssh: clones to the raspberry pi.
Help us debug the ssh: protocol when the server is a Windows machine.
Spend $5/month on a Linode or Droplet and host all your repos there, behind HTTPS. This gives you the added benefit of off-site backups.
(11) By Warren Young (wyetr) on 2019-09-24 20:57:34 in reply to 5 [link] [source]
My tip matters when the client is on the Windows side. I see on closer reading that your client is Linux instead, so it will use ssh
commands for establishing the tunnel, not plink
.
However, keep it in mind if you ever do Windows-to-Windows clones and sync.
(12) By ZanyScum (zany) on 2019-09-24 23:21:23 in reply to 9 [link] [source]
It is just not working.
The repository on Windows has 3MB, 1 text file and one image (jpg).
I started the server with
fossil server -port 18080 test.repo
I created a tunnel and used the following config:
Host desktop HostName 192.168.1.99 User MyName Port 22 ServerAliveInterval 60 LocalForward 18080 127.0.0.1:18080 ForwardX11 yesI connect via ssh
ssh desktop
and then in other terminal I do the cloning:
fossil clone http://localhost:18080 test02.repo Round-trips: 2 Artifacts sent: 0 received: 4 Clone done, sent: 469 received: 1365 ip: ::1 Rebuilding repository meta-data... 100.0% complete... Extra delta compression... Vacuuming the database... project-id: 3effc9d1ebdff561415fd287dce9c15177a94415 server-id: 6951077b24e0c386a2db10a03b1832fdcdb0ab30 admin-user: MyName (password is "adf13b")So I get a valid repository. I only has a size of 229376 byes (instead of 3MB).
I do a
fossil open ../test02.repo
project-name: <unnamed>
repository: /home/MyName/test/j/../test02.repo
local-root: /home/MyName/test/j/
config-db: /home/MyName/.fossil
project-code: 3effc9d1ebdff561415fd287dce9c15177a94415
check-ins: 0
There is no data. No file. Nothing. The clone command worked normal, without error or anything weird.
I am going to bed. :-(
(13) By Martin Gagnon (mgagnon) on 2019-09-25 12:35:12 in reply to 10 [link] [source]
- Help us debug the ssh: protocol when the server is a Windows machine.
I've set a ssh server on a windows-10 vm, and tried to clone it using --httptrace.
Then I did the same by cloning it with ssh://localhost/path/repo.fossil
(MacOS).
When comparing the http-reply-<n>.txt
files (converted in hex), I've notice that the clone done from the windows server contain an extra <0d>
caracter at the end of each line.
On the good one, lines end by <0d><0a>, versus <0d><0d><0a> on the bad one.
Also, I've notice that the first reply was trunkated the middle of the "# timestamps ...
" line and the rest of the line appears on the beginning of the next reply.
Here's my test repo with the httptraces files. https://drive.google.com/file/d/1ZglTjmGbz6GSHg-FgrkpOCZHt8c3ntCK/view?usp=sharing
(good/ is clone via ssh with macosx server while bad/ is via ssh with windows-10 server, the .xxd files are the files converted in hex using xxd utility)
(14) By Andy Bradford (andybradford) on 2019-09-25 13:36:45 in reply to 8 [link] [source]
> A possibility I am not so happy about even if it were possible, but > I'd like to know if it IS possible: Can I temporarily start my normal > no-server fossil as http server, connect via ssh and get the data from > localhost:8080 or whatever port I'd use? Yes, you should be able to do that easily provided you configure your Windows SSH server to allow tunnels. Just start up "fossil server" on localhost (maybe with the --localhost option), and then configure an SSH tunnel on your Linux machine like this: Assuming you have a directory named D:\repos that has a number of files named project1.fossil, project2.fossil, project3.fossil, etc... # setup a fossil server fossil server --localhost --port 8080 D:\repos # establish the SSH tunnel on your Linux machine ssh -L8080:localhost:8080 username@windowsmachine # now you can clone over that tunnel on your Linux machine: fossil clone http://localhost:8080/project1 Andy
(15) By Andy Bradford (andybradford) on 2019-09-25 13:47:06 in reply to 7 [link] [source]
> I don't know how to configure an ssh server on a Windows machine. That > is not something I've ever done before, so cloning off of a Windows > machine using ssh is not something I've ever tested. SSH on Windows is one of the new hot features for Windows (even though SSH has been available on Windows for some time using 3rd party SSH servers), so I imagine we'll start getting more questions about it. https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_server_configuration If I have some time, I'll try to get an environment setup for it and see what's wrong (unless someone beats me to it).
(16) By Andy Bradford (andybradford) on 2019-09-25 13:53:34 in reply to 13 [link] [source]
> On the good one, lines end by <0d><0a>, versus <0d><0d><0a> on the bad one. Yep, it looks like on Windows, for whatever reason, there is an extra carriage return (CR character) in the headers and also CR characters in the body that should not be there. So for some reason, something altered (or Fossil generated) the entire HTTP response, headers and body, and put CR at the end of every line. The reason why it's truncated is because the headers claim that the entire response is 750 bytes, which of course is not true given that there are now a lot more characters than expected due to the presence of all the CR characters. Content-Length: 750^M^M Thanks, ANdy
(17) By Richard Hipp (drh) on 2019-09-25 13:56:24 in reply to 1 [link] [source]
I got sshd.exe running on my Windows10 machine, and was therefore able to reproduce the problem and fix it. If you install check-in 28b15b4816b9e05b or later on the Windows side, then ssh: should work work now.
I ran the following command from my linux desktop:
fossil clone 'ssh://drh@192.168.NNN.NNN/c:/fossils/sqlite.fossil' sqlite.fossil
An IP address of the Windows10 laptop was substituted for 192.168.NNN.NNN, of course. I keep my Fossil repositories in the C:/Fossils directory on the windows machine. I was prompted for a password, and the clone ran to completion and gave me a correct and complete repository.
Related Issue:
For the ssh: client on Windows, the default name of the ssh command is "plink". But now that Microsoft actually makes a working "ssh" command available on Windows, I wonder if we should change the default to be "ssh"?
This came up while I was testing the new code in the other direction - Windows client to linux server. I had to add the "-c ssh" option to the "fossil clone" command-line on Windows in order to get the clone to work.
(18) By Andy Bradford (andybradford) on 2019-09-25 13:59:54 in reply to 12 [link] [source]
> It is just not working. Apparently there is something introducing extra carriage returns in the data, so not even tunnels will work. Just out of curiousity, which version of Fossil are you running on both Windows and Linux? Thanks, Andy
(19) By Andy Bradford (andybradford) on 2019-09-25 14:11:21 in reply to 17 [link] [source]
> But now that Microsoft actually makes a working "ssh" command > available on Windows, I wonder if we should change the default to be > "ssh"? Now that Windows has SSH, it might make sense to change the default, however, I have a question that I don't know the answer to yet regarding ssh-agent capabilities. With plink, it is possible that users will be using pagent and thus have the features that pagent provides. Does the new Windows SSH implementation come with an ssh-agent? Will a change to the default somehow break those who are currently relying on it? This may or may not influence the decision to switch the default from plink to ssh. Thanks, Andy
(20) By ZanyScum (zany) on 2019-09-25 18:47:54 in reply to 18 [link] [source]
Hi,
I just downloaded the Windows version yesterday: This is fossil version 2.9 [0fd79a3e09] 2019-07-13 13:05:19 UTC
With Debian 9 I just used what I got via apt-get This is fossil version 1.37 [463d2a7b49] 2017-01-18 12:25:23 UTC
After looking at this, I feel pretty stupid.
TESTING..........TESTING..........TESTING..........TESTING..........
IT WORKS!!!!
When I use the same version - the Windows AND Linux versions I downloaded yesterday, then I can clone the repository.
BUT: I can only open the cloned repository with the same version. I think the one from apt-get is just too old for the current format.
So....what I described in https://www.fossil-scm.org/forum/forumpost/eaf3ed51ae is working IF you use the same version on both sides.
I am sorry to not have seen this potential problem by myself.
Thanks Andy!
It works!!!!!!!!!!!!
(21.1) By ZanyScum (zany) on 2019-09-25 19:23:42 edited from 21.0 in reply to 20 [link] [source]
Now that we now the problem was with the versions, why not try it again with the simple ssh clone variant?
fossil clone 'ssh://desktop/D:/_ssh/repositories/test.repo?fossil=C:/myname/bin/win/32/fossil.exe' test.repo
the ssh:// URL is asking to run an unsafe command C:/myname/bin/win/32/fossil.exe on the server. Aborted
Still not working. Pretty sure it is silly me AGAIN :-( Do you guys see what is wrong?
(22) By Warren Young (wyoung) on 2019-09-25 19:25:29 in reply to 20 [link] [source]
With Debian 9 I just used what I got via apt-get
Debian 9 went out of support several months ago. You should upgrade to Debian 10, which ships Fossil 2.8.
IF you use the same version
The actual case is much less stringent than that: if you're using Fossil >= 2.1 on one side, you need to be using >= 2.1 on the other side, too, unless you go out of your way to allow use of Fossil 1.x.
Thus, using Fossil 2.9 on Windows with Debian 2.8 on Debian 10 is just fine.
(23) By ZanyScum (zany) on 2019-09-25 19:26:25 in reply to 17 [link] [source]
Sorry, I did not see this message. I'll try this soon.
(24) By Warren Young (wyoung) on 2019-09-25 19:29:50 in reply to 21.1 [link] [source]
Do you guys see what is wrong?
According to your prior reply here, you're using the 2.9 release version on Windows, which doesn't have the fix drh just made for you. That means you can either wait for Fossil 2.10, or build Fossil from source, or hope someone else builds a binary for you. Me, I'd either build Fossil from source or switch to the Cygwin build, which won't have this problem. WSL won't have this problem, either.
(25) By ZanyScum (zany) on 2019-09-25 19:30:12 in reply to 23 [link] [source]
Ah, sorry, do I need to build this myself or can I download the binary somehow?
(26.2) By ZanyScum (zany) on 2019-09-25 19:53:01 edited from 26.1 in reply to 25 [link] [source]
If my fossil under Windows tells me
D:\>fossil version
This is fossil version 2.10 [28b15b4816] 2019-09-25 13:45:15 UTC
Then I have the right version? Still not working.
Something wrong with
fossil clone -v ssh://pcname/D:/_ssh/repositories/test.repo?fossil=C:/myname/bin/win/32/fossil.exe test2.repo
?
Btw, is there a way to use the "Fossil Wiki" markup and quote for more than one line at once?
Update:
fossil clone 'ssh://desktop/D:/_ssh/repositories/test.repo?fossil=C:/myname/bin/win/32/fossil.exe' test2.repo
does not work either.
(27) By Andy Bradford (andybradford) on 2019-09-25 19:51:12 in reply to 26.0 [link] [source]
Should that perhaps be: fossil clone -v ssh://pcname//D:/_ssh/repositories/test.repo?fossil=C:/andreas/bin/win/32/fossil.exe test2.repo With SSH, all paths are relative to home directory unless they begin with / and the first / in the URL is part of the URL, not part of the path on the OS. I don't know if that's how it works on Windows, but that's how it works with SSH everywhere else. Otherwise, what output do you get when you try? Thanks, Andy
(28) By Andy Bradford (andybradford) on 2019-09-25 19:55:16 in reply to 20 [link] [source]
Some of the card formats in the sync protocol have changed between 1.37 and 2.9, so even if you were able to get around the extra carriage returns, it still would have failed to clone due to it not understanding the new hash algorithm used in the cards (e.g. cfile, igot, etc...). Thanks, Andy
(29) By ZanyScum (zany) on 2019-09-25 20:07:23 in reply to 27 [link] [source]
I am sorry, I do get a bit confused with the way I have to check for new answers o n the whole forum page.
The reply I get with //D: and /D: is the same:
I removed the paths to remove potential problems. I now have the .exe and the .repo in the root of D:.
fossil clone -v ssh://desktop/D:/test.repo?fossil=D:/fossil.exe test2.repo
Bytes Cards Artifacts Deltas
waiting for server...
the ssh:// URL is asking to run an unsafe command D:/fossil.exe on the server.
Aborted
fossil clone -v ssh://desktop//D:/test.repo?fossil=D://fossil.exe test2.repo
Bytes Cards Artifacts Deltas
waiting for server...
the ssh:// URL is asking to run an unsafe command D://fossil.exe on the server.
Aborted
I really fear it is something incredible stupid on my side, something you don't even think about.
This is the version output:
$fossil version
This is fossil version 2.9 [0fd79a3e09] 2019-07-13 13:05:19 UTC
$ ssh desktop 'D:\fossil version'
This is fossil version 2.10 [28b15b4816] 2019-09-25 13:45:15 UTC
(30) By ZanyScum (zany) on 2019-09-25 20:12:32 in reply to 29 [link] [source]
ARG!!! It works!
fossil clone -v ssh://desktop/D:/test.repo?fossil=D:/fossil test2.repo
Bytes Cards Artifacts Deltas
Sent: 81 1 0 0
Received: 2984901 15 6 0
Sent: 86 2 0 0
Received: 808 9 0 0
Clone done, sent: 557 received: 2985672 ip: desktop
Rebuilding repository meta-data...
100.0% complete...
Extra delta compression...
Vacuuming the database...
project-id: 3effc9d1ebdff561415fd287dce9c15177a94415
server-id: f2ab377681cf001a70fef0d5bfe7c9be6ac6ed40
admin-user: myname (password is "b24cf2")
I had to use D:/ not D:// and I had to remove the .exe from the path to the fossil executable on Windows. Then it works.
It works!
Thanks to all for helping me! And sorry if I was a bit slow some (all the) time.
Thanks guys!
(31) By Martin Gagnon (mgagnon) on 2019-09-25 20:59:01 in reply to 30 [link] [source]
the ssh:// URL is asking to run an unsafe command D://fossil.exe on the server.
...
I had to remove the .exe from the path to the fossil executable on Windows
FYI: This got fixed today here 8f70ccaac82d0e30. This is a fix for the client side.
(32) By ZanyScum (zany) on 2019-09-26 09:14:17 in reply to 31 [link] [source]
Thank you for the information.
(33) By Andy Bradford (andybradford) on 2019-09-26 13:15:48 in reply to 29 [link] [source]
said zany: > I am sorry, I do get a bit confused with the way I have to check for > new answers o n the whole forum page. This is why I often make use of the "Chronological" button at the top of the page. It's much easier to keep track of new items this way. Andy