Fossil Forum

Clone via ssh from Windows drive D: to Linux?
Login

Clone via ssh from Windows drive D: to Linux?

Clone via ssh from Windows drive D: to Linux?

(1) By ZanyScum (zany) on 2019-09-24 15:27:31 [link] [source]

What I have:

  • I have fossil on a Windows 10 system.
  • I have on this Windows system a ssh server running and able to be connected to via ssh keys.
  • I have a Linux system which can ssh into the Windows system.
  • I can connect using the config file, so "ssh pcname" is working

What I want:

  • I'd like to use the fossil clone command from Linux on a repository which resides on D: on a Windows system.

My tries so far:

  • fossil clone ssh://pcname/D:\_ssh\repositories\test.repo test.repo
  • fossil clone ssh://pcname/D:/_ssh/repositories/test.repo test.repo
  • fossil clone ssh://pcname//D:/_ssh/repositories/test.repo test.repo

What I get: server did not reply Clone done, sent: 280 received: 81 ip: pcname server returned an error - clone aborted

How do I do it the right way?

(2) By Richard Hipp (drh) on 2019-09-24 15:38:54 in reply to 1 [link] [source]

Seems like the second approach is correct, and it seems to have worked, did it not? What makes you think it didn't work?

The way clone (or sync) via SSH works is that it uses ssh to run the following command on the remote system:

fossil test-http REPONAME

Where REPONAME is the pathname of the ssh: URI. You want D:/_ssh/repositories/test.repo and so I think you want the second URI. The input and output to this command is the ordinary HTTP traffic that is normally sent over a network socket, though in this case it travels over the SSH connection instead.

The ssh command expects fossil to be on %PATH%. If it is not, you will need to use a query parameter to specify the Fossil binary:

 ssh://pcname/D:/_ssh/repositories/test.repo?fossil=c:/what/ever/fossil.exe

But I don't think you need to do this because apparently your second case worked, at least to judge from the output you got back.

(3.1) By Warren Young (wyetr) on 2019-09-24 17:20:11 edited from 3.0 in reply to 1 [source]

On Windows, Fossil calls "plink -ssh -T" to set up SSH tunnels by default, which relies on either the plink.exe from PuTTY or the one from SSH Communications. If you don't want to install plink.exe from either source, but instead want to use OpenSSH — perhaps the version recently added to Windows — you have to override this with the ssh-command setting:

 c:\> fossil set ssh-command "ssh -e none -T"

ssh.exe will need to be in the %PATH%, of course.

You might want to make this setting global, so it affects all repos on that system.

(4) By ZanyScum (zany) on 2019-09-24 18:13:16 in reply to 2 [link] [source]

You directed me in the right direction. After connecting via ssh fossil was not in the path.

The following seems to do ... something:

fossil clone ssh://pcname/D:/_ssh/repositories/test.repo?fossil=C:/myname/bin/win/32/fossil.exe test2.repo

It creates test2.repo with a size of 217088 bytes (versus the original 3MB) then it ... don't know... stops? It keeps running, I can enter text and see it and press return, but it is is like the program is still running or waiting for a special input.

Or if the command above is still missing something.

(5) By ZanyScum (zany) on 2019-09-24 18:32:08 in reply to 3.1 [link] [source]

I am sorry, I am not sure what you are talking about.

On Windows I am calling a ssh server. I do not work with putty, plink or any ssh client at all.

(6) By ZanyScum (zany) on 2019-09-24 18:41:52 in reply to 4 [link] [source]

I just used the -v option:

fossil clone -v ssh://desktop/D:/_ssh/repositories/test.repo?fossil=C:/andreas/bin/win/32/fossil.exe test2.repo
Bytes Cards Artifacts Deltas
waiting for server...

(7) By Richard Hipp (drh) on 2019-09-24 18:58:59 in reply to 4 [link] [source]

I don't know how to configure an ssh server on a Windows machine. That is not something I've ever done before, so cloning off of a Windows machine using ssh is not something I've ever tested. I've done it the other way around - Windows client and unix server - but I've never tried to clone onto a unix client from a Windows server.

Perhaps there is some \r\n translation issues messing up the protocol. The messages that go back and forth from client to server contain byte-counts, and so if extra \r characters get inserted or removed by SSH, then things will not go well. We've worked through all those issues for a Windows client hitting a unix server, but not the other way around.

Or it could be something else. I don't have any way of testing.

Have you considered just running "fossil server" on the Windows machine, then cloning or syncing using ordinary http:?

(8) By ZanyScum (zany) on 2019-09-24 19:38:19 in reply to 7 [link] [source]

Thank you for trying to help.

My situation is the following: I am using privately svn since 15 years or more. Of course not many more ;-)

Now I am trying to go away from 2x desktop + 1x netbook + 1x server (24/7). I want to just remove my Jenkins/Nagios/Subversion/Reddit server and just have a repository on my desktop and another on on my Netbook, perhaps later on a laptop too. So... no server of any kind.

I hate git. I have used it for ca. 3 years on a daily basis and I have learned to really hate it.

So after some research I am currently thinking of using either fossil or mercurial. But copy-one-file-and-your-installation-is-done is such an unbeatable argument, I just HAD to try it.

Now the reason for my wall of text: I will perhaps have to sync different systems via WLan and I will not sync my personal data over WLan and http. Even https would be something I'd like to avoid. I just trust ssh more. No valid reason. It is just that way.

That is the reason I do not want to have fossil running as http server.

A possibility I am not so happy about even if it were possible, but I'd like to know if it IS possible: Can I temporarily start my normal no-server fossil as http server, connect via ssh and get the data from localhost:8080 or whatever port I'd use?

Http over ssh? If it is possible, how I start the http service (Windows and Linux please) and how would the connection line look like?

(9) By ZanyScum (zany) on 2019-09-24 19:40:11 in reply to 8 [link] [source]

Sorry, I am stupid, that is what ssh tunnels are for. I'll try it perhaps the next days.

(10) By Richard Hipp (drh) on 2019-09-24 20:37:31 in reply to 8 [link] [source]

Other options to consider:

  1. Put Fossil on a raspberry pi sitting somewhere in the corner, and use ssh: clones to the raspberry pi.

  2. Help us debug the ssh: protocol when the server is a Windows machine.

  3. Spend $5/month on a Linode or Droplet and host all your repos there, behind HTTPS. This gives you the added benefit of off-site backups.

(11) By Warren Young (wyetr) on 2019-09-24 20:57:34 in reply to 5 [link] [source]

My tip matters when the client is on the Windows side. I see on closer reading that your client is Linux instead, so it will use ssh commands for establishing the tunnel, not plink.

However, keep it in mind if you ever do Windows-to-Windows clones and sync.

(12) By ZanyScum (zany) on 2019-09-24 23:21:23 in reply to 9 [link] [source]

It is just not working.

The repository on Windows has 3MB, 1 text file and one image (jpg).

I started the server with

fossil server -port 18080  test.repo

I created a tunnel and used the following config:

  Host desktop
        HostName                192.168.1.99
        User                    MyName
        Port                    22
        ServerAliveInterval     60
        LocalForward            18080 127.0.0.1:18080
        ForwardX11              yes
I connect via ssh

ssh desktop

and then in other terminal I do the cloning:

  fossil clone http://localhost:18080 test02.repo
  Round-trips: 2   Artifacts sent: 0  received: 4
  Clone done, sent: 469  received: 1365  ip: ::1
  Rebuilding repository meta-data...
  100.0% complete...
  Extra delta compression...
  Vacuuming the database...
  project-id: 3effc9d1ebdff561415fd287dce9c15177a94415
  server-id:  6951077b24e0c386a2db10a03b1832fdcdb0ab30
  admin-user: MyName (password is "adf13b")
So I get a valid repository. I only has a size of 229376 byes (instead of 3MB).

I do a

  fossil open ../test02.repo
  project-name: <unnamed>
  repository:   /home/MyName/test/j/../test02.repo
  local-root:   /home/MyName/test/j/
  config-db:    /home/MyName/.fossil
  project-code: 3effc9d1ebdff561415fd287dce9c15177a94415
  check-ins:    0
There is no data. No file. Nothing. The clone command worked normal, without error or anything weird.

I am going to bed. :-(

(13) By Martin Gagnon (mgagnon) on 2019-09-25 12:35:12 in reply to 10 [link] [source]

  1. Help us debug the ssh: protocol when the server is a Windows machine.

I've set a ssh server on a windows-10 vm, and tried to clone it using --httptrace.

Then I did the same by cloning it with ssh://localhost/path/repo.fossil (MacOS).

When comparing the http-reply-<n>.txt files (converted in hex), I've notice that the clone done from the windows server contain an extra <0d> caracter at the end of each line.

On the good one, lines end by <0d><0a>, versus <0d><0d><0a> on the bad one.

Also, I've notice that the first reply was trunkated the middle of the "# timestamps ..." line and the rest of the line appears on the beginning of the next reply.

Here's my test repo with the httptraces files. https://drive.google.com/file/d/1ZglTjmGbz6GSHg-FgrkpOCZHt8c3ntCK/view?usp=sharing

(good/ is clone via ssh with macosx server while bad/ is via ssh with windows-10 server, the .xxd files are the files converted in hex using xxd utility)

(14) By Andy Bradford (andybradford) on 2019-09-25 13:36:45 in reply to 8 [link] [source]

> A possibility I  am not so happy  about even if it  were possible, but
> I'd like to know if it IS  possible: Can I temporarily start my normal
> no-server fossil as http server, connect via ssh and get the data from
> localhost:8080 or whatever port I'd use?

Yes, you  should be able to  do that easily provided  you configure your
Windows SSH  server to allow tunnels.  Just start up "fossil  server" on
localhost (maybe with the --localhost option), and then configure an SSH
tunnel on your Linux machine like this:

Assuming you have a directory named  D:\repos that has a number of files
named project1.fossil, project2.fossil, project3.fossil, etc...

# setup a fossil server
fossil server --localhost --port 8080 D:\repos

# establish the SSH tunnel on your Linux machine
ssh -L8080:localhost:8080 username@windowsmachine

# now you can clone over that tunnel on your Linux machine:
fossil clone http://localhost:8080/project1

Andy

(15) By Andy Bradford (andybradford) on 2019-09-25 13:47:06 in reply to 7 [link] [source]

> I don't know how to configure an ssh server on a Windows machine. That
> is not  something I've ever done  before, so cloning off  of a Windows
> machine using ssh is not something I've ever tested.

SSH on Windows is  one of the new hot features  for Windows (even though
SSH has  been available  on Windows  for some time  using 3rd  party SSH
servers), so I imagine we'll start getting more questions about it.

https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_server_configuration

If I have some time, I'll try to get an environment setup for it and see
what's wrong (unless someone beats me to it).

(16) By Andy Bradford (andybradford) on 2019-09-25 13:53:34 in reply to 13 [link] [source]

> On the good one, lines end by <0d><0a>, versus <0d><0d><0a> on the bad one.

Yep, it  looks like on Windows,  for whatever reason, there  is an extra
carriage return (CR character) in the  headers and also CR characters in
the body that should not be there.

So for some  reason, something altered (or Fossil  generated) the entire
HTTP response, headers  and body, and put  CR at the end  of every line.
The reason  why it's  truncated is  because the  headers claim  that the
entire response  is 750 bytes,  which of course  is not true  given that
there are now a lot more characters than expected due to the presence of
all the CR characters.

Content-Length: 750^M^M

Thanks,

ANdy

(17) By Richard Hipp (drh) on 2019-09-25 13:56:24 in reply to 1 [link] [source]

I got sshd.exe running on my Windows10 machine, and was therefore able to reproduce the problem and fix it. If you install check-in 28b15b4816b9e05b or later on the Windows side, then ssh: should work work now.

I ran the following command from my linux desktop:

fossil clone 'ssh://drh@192.168.NNN.NNN/c:/fossils/sqlite.fossil' sqlite.fossil

An IP address of the Windows10 laptop was substituted for 192.168.NNN.NNN, of course. I keep my Fossil repositories in the C:/Fossils directory on the windows machine. I was prompted for a password, and the clone ran to completion and gave me a correct and complete repository.

Related Issue:

For the ssh: client on Windows, the default name of the ssh command is "plink". But now that Microsoft actually makes a working "ssh" command available on Windows, I wonder if we should change the default to be "ssh"?

This came up while I was testing the new code in the other direction - Windows client to linux server. I had to add the "-c ssh" option to the "fossil clone" command-line on Windows in order to get the clone to work.

(18) By Andy Bradford (andybradford) on 2019-09-25 13:59:54 in reply to 12 [link] [source]

> It is just not working.

Apparently there is something introducing  extra carriage returns in the
data, so not even tunnels will work.

Just out of curiousity, which version  of Fossil are you running on both
Windows and Linux?

Thanks,

Andy

(19) By Andy Bradford (andybradford) on 2019-09-25 14:11:21 in reply to 17 [link] [source]

> But  now  that  Microsoft  actually  makes  a  working  "ssh"  command
> available on Windows,  I wonder if we should change  the default to be
> "ssh"?

Now that  Windows has SSH,  it might make  sense to change  the default,
however, I have a question that I don't know the answer to yet regarding
ssh-agent capabilities.  With plink, it  is possible that users  will be
using pagent and  thus have the features that pagent  provides. Does the
new Windows SSH implementation come with an ssh-agent?

Will  a change  to the  default somehow  break those  who are  currently
relying on it?

This may  or may not influence  the decision to switch  the default from
plink to ssh.

Thanks,

Andy

(20) By ZanyScum (zany) on 2019-09-25 18:47:54 in reply to 18 [link] [source]

Hi,

I just downloaded the Windows version yesterday: This is fossil version 2.9 [0fd79a3e09] 2019-07-13 13:05:19 UTC

With Debian 9 I just used what I got via apt-get This is fossil version 1.37 [463d2a7b49] 2017-01-18 12:25:23 UTC

After looking at this, I feel pretty stupid.

TESTING..........TESTING..........TESTING..........TESTING..........

IT WORKS!!!!

When I use the same version - the Windows AND Linux versions I downloaded yesterday, then I can clone the repository.

BUT: I can only open the cloned repository with the same version. I think the one from apt-get is just too old for the current format.

So....what I described in https://www.fossil-scm.org/forum/forumpost/eaf3ed51ae is working IF you use the same version on both sides.

I am sorry to not have seen this potential problem by myself.

Thanks Andy!

It works!!!!!!!!!!!!

(21.1) By ZanyScum (zany) on 2019-09-25 19:23:42 edited from 21.0 in reply to 20 [link] [source]

Now that we now the problem was with the versions, why not try it again with the simple ssh clone variant?

fossil clone 'ssh://desktop/D:/_ssh/repositories/test.repo?fossil=C:/myname/bin/win/32/fossil.exe' test.repo
the ssh:// URL is asking to run an unsafe command C:/myname/bin/win/32/fossil.exe on the server. Aborted

Still not working. Pretty sure it is silly me AGAIN :-( Do you guys see what is wrong?

(22) By Warren Young (wyoung) on 2019-09-25 19:25:29 in reply to 20 [link] [source]

With Debian 9 I just used what I got via apt-get

Debian 9 went out of support several months ago. You should upgrade to Debian 10, which ships Fossil 2.8.

IF you use the same version

The actual case is much less stringent than that: if you're using Fossil >= 2.1 on one side, you need to be using >= 2.1 on the other side, too, unless you go out of your way to allow use of Fossil 1.x.

Thus, using Fossil 2.9 on Windows with Debian 2.8 on Debian 10 is just fine.

(23) By ZanyScum (zany) on 2019-09-25 19:26:25 in reply to 17 [link] [source]

Sorry, I did not see this message. I'll try this soon.

(24) By Warren Young (wyoung) on 2019-09-25 19:29:50 in reply to 21.1 [link] [source]

Do you guys see what is wrong?

According to your prior reply here, you're using the 2.9 release version on Windows, which doesn't have the fix drh just made for you. That means you can either wait for Fossil 2.10, or build Fossil from source, or hope someone else builds a binary for you. Me, I'd either build Fossil from source or switch to the Cygwin build, which won't have this problem. WSL won't have this problem, either.

(25) By ZanyScum (zany) on 2019-09-25 19:30:12 in reply to 23 [link] [source]

Ah, sorry, do I need to build this myself or can I download the binary somehow?

(26.2) By ZanyScum (zany) on 2019-09-25 19:53:01 edited from 26.1 in reply to 25 [link] [source]

If my fossil under Windows tells me

D:\>fossil version
This is fossil version 2.10 [28b15b4816] 2019-09-25 13:45:15 UTC

Then I have the right version? Still not working.

Something wrong with

fossil clone -v ssh://pcname/D:/_ssh/repositories/test.repo?fossil=C:/myname/bin/win/32/fossil.exe test2.repo

?

Btw, is there a way to use the "Fossil Wiki" markup and quote for more than one line at once?

Update:

fossil clone 'ssh://desktop/D:/_ssh/repositories/test.repo?fossil=C:/myname/bin/win/32/fossil.exe' test2.repo

does not work either.

(27) By Andy Bradford (andybradford) on 2019-09-25 19:51:12 in reply to 26.0 [link] [source]

Should that perhaps be:

fossil clone -v ssh://pcname//D:/_ssh/repositories/test.repo?fossil=C:/andreas/bin/win/32/fossil.exe test2.repo

With SSH,  all paths are  relative to  home directory unless  they begin
with / and  the first / in the URL  is part of the URL, not  part of the
path on  the OS. I  don't know  if that's how  it works on  Windows, but
that's how it works with SSH everywhere else.

Otherwise, what output do you get when you try?

Thanks,

Andy

(28) By Andy Bradford (andybradford) on 2019-09-25 19:55:16 in reply to 20 [link] [source]

Some of the card formats in  the sync protocol have changed between 1.37
and 2.9,  so even  if you  were able  to get  around the  extra carriage
returns, it still would have failed to clone due to it not understanding
the new hash algorithm used in the cards (e.g. cfile, igot, etc...).

Thanks,

Andy

(29) By ZanyScum (zany) on 2019-09-25 20:07:23 in reply to 27 [link] [source]

I am sorry, I do get a bit confused with the way I have to check for new answers o n the whole forum page.

The reply I get with //D: and /D: is the same:

I removed the paths to remove potential problems. I now have the .exe and the .repo in the root of D:.

fossil clone -v ssh://desktop/D:/test.repo?fossil=D:/fossil.exe test2.repo
Bytes Cards Artifacts Deltas
waiting for server...
the ssh:// URL is asking to run an unsafe command D:/fossil.exe on the server.
Aborted
fossil clone -v ssh://desktop//D:/test.repo?fossil=D://fossil.exe test2.repo
Bytes Cards Artifacts Deltas
waiting for server...
the ssh:// URL is asking to run an unsafe command D://fossil.exe on the server.
Aborted

I really fear it is something incredible stupid on my side, something you don't even think about.

This is the version output:

$fossil version
This is fossil version 2.9 [0fd79a3e09] 2019-07-13 13:05:19 UTC
$ ssh desktop 'D:\fossil version'
This is fossil version 2.10 [28b15b4816] 2019-09-25 13:45:15 UTC

(30) By ZanyScum (zany) on 2019-09-25 20:12:32 in reply to 29 [link] [source]

ARG!!! It works!

fossil clone -v ssh://desktop/D:/test.repo?fossil=D:/fossil test2.repo
Bytes Cards Artifacts Deltas
Sent: 81 1 0 0
Received: 2984901 15 6 0
Sent: 86 2 0 0
Received: 808 9 0 0
Clone done, sent: 557 received: 2985672 ip: desktop
Rebuilding repository meta-data...
100.0% complete...
Extra delta compression...
Vacuuming the database...
project-id: 3effc9d1ebdff561415fd287dce9c15177a94415
server-id: f2ab377681cf001a70fef0d5bfe7c9be6ac6ed40
admin-user: myname (password is "b24cf2")

I had to use D:/ not D:// and I had to remove the .exe from the path to the fossil executable on Windows. Then it works.

It works!

Thanks to all for helping me! And sorry if I was a bit slow some (all the) time.

Thanks guys!

(31) By Martin Gagnon (mgagnon) on 2019-09-25 20:59:01 in reply to 30 [link] [source]

the ssh:// URL is asking to run an unsafe command D://fossil.exe on the server.

...

I had to remove the .exe from the path to the fossil executable on Windows

FYI: This got fixed today here 8f70ccaac82d0e30. This is a fix for the client side.

(32) By ZanyScum (zany) on 2019-09-26 09:14:17 in reply to 31 [link] [source]

Thank you for the information.

(33) By Andy Bradford (andybradford) on 2019-09-26 13:15:48 in reply to 29 [link] [source]

said zany:

> I am sorry, I  do get a bit confused with the way  I have to check for
> new answers o n the whole forum page.

This is why I often make use of the "Chronological" button at the top of
the page. It's much easier to keep track of new items this way.

Andy