Make sure that we cleanup remainders when the handle is closed and when
the subsystem is shutdown.
Existing flaw: if a single handle sends packets that haven't been
replied to yet at the time when the handle is closed, those packets will
arrive later and end up in the generic packet brigade queue and they
will remain in there until flushed. They will use unnecessary memory,
make things slower and they will ruin the SFTP handling if the
request_id counter ever wraps (highly unlikely to every happen).
The SFTP read function now does transfers the same way the SFTP write
function was made to recently: it creates a list of many outgoing
FXP_READ packets that each asks for a small data chunk. The code then
tries to keep sending read request while collecting the acks for the
previous requests and returns the received data.
The loop that waits for remote.close to get set may end up looping
forever since session->socket_state gets set to
LIBSSH2_SOCKET_DISCONNECTED by the packet_add() function called from the
transport_read() function and after having been set to
LIBSSH2_SOCKET_DISCONNECTED, the transport_read() function will only
return 0.
Bug: http://trac.libssh2.org/ticket/198
Split off libssh2_sftp_seek64 from the libssh2_sftp_seek man page, and
mentioned that we consider the latter deprecated. Also added a mention
about the dangers of doing seek during writing or reading.
The new SFTP write code caused a regression as the seek function no
longer worked as it didn't set the write position properly.
It should be noted that seeking is STRONGLY PROHIBITED during upload, as
the upload magic uses two different offset positions and the multiple
outstanding packets etc make them sensitive to change in the midst of
operations.
This functionality was just verified with the new example code
sftp_append. This bug was filed as bug #202:
Bug: http://trac.libssh2.org/ticket/202
I ran SFTP upload tests against localhost. It showed that to make the
app reach really good speeds, I needed to do a little code tweak and
change MAX_SFTP_OUTGOING_SIZE from 4000 to 30000. The tests I did before
with the high latency tests didn't show any real difference whatever I
had that size set to.
This number is the size in bytes that libssh2 cuts off the large input
buffer and sends off as an individual sftp packet.
This is an example that is very similar to sftp_write_nonblock.c, with
the exception that this uses
1 - a larger upload buffer
2 - a sliding buffer mechnism to allow the app to keep sending lots of
data to libssh2 without having to first drain the buffer.
These are two key issues to make libssh2 SFTP uploads really perform
well at this point in time.
The attempts made to have _libssh2_channel_write() accept larger pieces
of data and split up the data by itself into 32700 byte chunks and pass
them on to channel_write() in a loop as a way to do faster operations on
larger data blocks was a failed attempt.
The reason why it is difficult:
The API only allows EAGAIN or a length to be returned. When looping over
multiple blocks to get sent, one block can get sent and the next might
not. And yet: when transport_send() has returned EAGAIN we must not call
it again with new data until it has returned OK on the existing data it
is still working on. This makes it a mess and we do get a much easier
job by simply returning the bytes or EAGAIN at once, as in the EAGAIN
case we can assume that we will be called with the same arguments again
and transport_send() will be happy.
Unfortunately, I think we take a small performance hit by not being able
to do this.
This is a new example snippet. The code is largely based on ssh2_exec,
and is written by Tommy Lindgren. I edited it into C90 compliance and to
conform to libssh2 indent style and some more.
When a piece of data is sent from the send_existing() function we must
make the parent function return afterwards. Otherwise we risk that the
parent function tries to send more data and ends up getting an EGAIN for
that more data and since it can only return one return code it doesn't
return info for the successfully sent data.
As this change is a regression I now added a larger comment explaining
why it has to work like this.
Starting now, we unconditionally use the internal replacement functions
for send() and recv() - creatively named _libssh2_recv() and
_libssh2_send().
On errors, these functions return the negative 'errno' value instead of
the traditional -1. This design allows systems that have no "natural"
errno support to not have to invent it. It also means that no code
outside of these two transfer functions should use the errno variable.
Some checks are better done in _libssh2_channel_write just once per
write instead of in channel_write() since the looping will call the
latter function multiple times per _libssh2_channel_write() invoke.
The SFTP handle struct now buffers number of acked bytes that haven't
yet been returned. The way this is used is as following:
1. sftp_write() gets called with a buffer of let say size 32000. We
split 32000 into 8 smaller packets and send them off one by one. One of
them gets acked before the function returns so 4000 is returned.
2. sftp_write() gets called again a short while after the previous one,
now with a much smaller size passed in to the function. Lets say 8000.
In the mean-time, all of the remaining packets from the previous call
have been acked (7*4000 = 28000). This function then returns 8000 as all
data passed in are already sent and it can't return any more than what
it got passed in. But we have 28000 bytes acked. We now store the
remaining 20000 in the handle->u.file.acked struct field to add up in
the next call.
3. sftp_write() gets called again, and now there's a backlogged 20000
bytes to return as fine and that will get skipped from the beginning
of the buffer that is passed in.
When SCP send or recv fails, it gets a special message from the server
with a warning or error message included. We have no current API to
expose that message but the foundation is there. Removed unnecessary use
of session struct fields.
I added size checks in several places. I fixed the code flow to be easier
to read in some places.
I removed unnecessary zeroing of structs. I removed unused struct fields.
We don't like magic numbers in the code. Now the acceptable failure
codes sent in the SSH_MSG_CHANNEL_OPEN_FAILURE message are added as
defined values in the private header file.
This function now only returns EAGAIN if a lower layer actually returned
EAGAIN to it. If nothing was acked and no EAGAIN was received, it will
now instead return 0.
If _libssh2_wait_socket() gets called but there's no direction set to
wait for, this causes a "hang". This code now detects this situation,
set a 1 second timeout instead and outputs a debug output about it.
SFTP packets come as [32 bit length][payload] and the code didn't
previously handle that the initial 32 bit field was read only partially
when it was read.
While setting up the session, ssh tries to determine the type of
encryption method it can use for the session. This requires looking at
the keys offered by the remote host and comparing these with the methods
supported by libssh2 (rsa & dss). To do this there is an iteration over
the array containing the methods supported by libssh2.
If there is no agreement on the type of encryption we come to the 3rd
entry of the hostkeyp array. Here hostkeyp is valid but *hostkep is
NULL. Thus when we dereference that in (*hostkeyp)->name there is a
crash
