Home » Community » Newbie corner » [Solved] SFTPBrowser example build failed (SFTPBrowser example buil error: error LNK2019: unresolved external symbol __imp_CertCloseStore referenced in function winstore_close)
| [Solved] SFTPBrowser example build failed [message #61263] |
Wed, 18 December 2024 10:10  |
 |
Scott_Huang
Messages: 27
Registered: December 2024
|
Promising Member |
|
|
I try sample SFTPBrowser since plan study the sample and then create a GUI for secure FTP later.
I try build with MSVS22x64 and CLANGx64 but both failed with below error message.
Do someone have experiences to fix this link issue? Thanks.
Build Error Message under MSVS22x64:
(): Linking has failed
(): crypto.lib(libdefault-lib-winstore_store.obj) : error LNK2019: unresolved external symbol __imp_CertCloseStore referenced in function winstore_close
(): crypto.lib(libdefault-lib-winstore_store.obj) : error LNK2019: unresolved external symbol __imp_CertFindCertificateInStore referenced in function winstore_win_advance
(): crypto.lib(libdefault-lib-winstore_store.obj) : error LNK2019: unresolved external symbol __imp_CertFreeCertificateContext referenced in function winstore_close
(): crypto.lib(libdefault-lib-winstore_store.obj) : error LNK2019: unresolved external symbol __imp_CertOpenSystemStoreW referenced in function winstore_open
(): C:\upp\out\MyApps\MSVS22x64.Gui\SFtpBrowser.exe : fatal error LNK1120: 4 unresolved externals
CLANGx64 error msg:
(): Linking has failed
(): ld.lld: error: undefined symbol: __declspec(dllimport) CertOpenSystemStoreW
(): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_ope n)
():
(): ld.lld: error: undefined symbol: __declspec(dllimport) CertFreeCertificateContext
(): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_ope n)
(): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_set _ctx_params)
(): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_clo se)
():
(): ld.lld: error: undefined symbol: __declspec(dllimport) CertFindCertificateInStore
(): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_set _ctx_params)
(): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_loa d)
():
(): ld.lld: error: undefined symbol: __declspec(dllimport) CertCloseStore
(): >>> referenced by libcrypto.a(libdefault-lib-winstore_store.obj):(winstore_clo se)
(): clang-18: error: linker command failed with exit code 1 (use -v to see invocation)
20241222 Update: This issue finally solved with Oblivion's professional and kindly support. You can go through all posts to get details or direct go to last post to get summary.
[Updated on: Sun, 22 December 2024 13:41] Report message to a moderator |
|
|
|
|
|
| Re: SFTPBrowser example build failed [message #61271 is a reply to message #61264] |
Wed, 18 December 2024 13:54  |
|
Scott_Huang
Messages: 27
Registered: December 2024
|
Promising Member |
|
|
Thank you @Oblivion. Yes, can successful build now.
And I try run the exe, it able connect to SFTP server.
However several issues:
1. My SFTP server have some folders, but the GUI did not show up.
2. I try create a folder through exe, no response & error. I use other software to connect to my SFTP, the folder does created.
3. I try upload a small text file, such as test.txt, only 47byte, it seems error. However, it create a 0 byte test.txt file in SFTP server if I use other software to checking.
4. So, I try to rebuild all, and hope the issue can be fixed since we just adjust SSH build way, the issue still same.
Below are the logs:
SSH: Session, oid: 1: Using Upp's memory managers.
SSH: Session, oid: 1: Session successfully initialized.
SSH: Session, oid: 1: Compression is disabled.
SSH: Session, oid: 1: Handshake successful.
SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive]
SSH: Session, oid: 1: Client succesfully authenticated.
SSH: SFtp, oid: 2: Session successfully initialized.
SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: /
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file
SSH: SFtp, oid: 2: Directory '//temp' is succesfully created.
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file
SSH: SFtp, oid: 3: Session successfully initialized.
SSH: SFtp, oid: 3: File '//test.txt' is successfully opened.
SSH: SFtp, oid: 3: File attributes successfully retrieved.
SSH: SFtp, oid: 3: EOF received.
SSH: SFtp, oid: 3: Stream write error.
SSH: SFtp, oid: 3: File handle freed.
SSH: SFtp, oid: 3: Session deinitalized.
SSH: SFtp, oid: 2: Session deinitalized.
SSH: Session, oid: 1: Successfully disconnected from the server.
SSH: Session, oid: 1: Session handles freed.
|
|
|
|
| Re: SFTPBrowser example build failed [message #61272 is a reply to message #61271] |
Wed, 18 December 2024 14:11 |
Oblivion
Messages: 1266
Registered: August 2007
|
Senior Contributor |
|
|
Hello Scott,
Quote:Thank you @Oblivion. Yes, can successful build now.
And I try run the exe, it able connect to SFTP server.
However several issues:
1. My SFTP server have some folders, but the GUI did not show up.
2. I try create a folder through exe, no response & error. I use other software to connect to my SFTP, the folder does created.
3. I try upload a small text file, such as test.txt, only 47byte, it seems error. However, it create a 0 byte test.txt file in SFTP server if I use other software to checking.
index.php?t=getfile&id=7022&private=0
4. So, I try to rebuild all, and hope the issue can be fixed since we just adjust SSH build way, the issue still same.
Below are the logs:
SSH: Session, oid: 1: Using Upp's memory managers.
SSH: Session, oid: 1: Session successfully initialized.
SSH: Session, oid: 1: Compression is disabled.
SSH: Session, oid: 1: Handshake successful.
SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive]
SSH: Session, oid: 1: Client succesfully authenticated.
SSH: SFtp, oid: 2: Session successfully initialized.
SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: /
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file
SSH: SFtp, oid: 2: Directory '//temp' is succesfully created.
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file
SSH: SFtp, oid: 3: Session successfully initialized.
SSH: SFtp, oid: 3: File '//test.txt' is successfully opened.
SSH: SFtp, oid: 3: File attributes successfully retrieved.
SSH: SFtp, oid: 3: EOF received.
SSH: SFtp, oid: 3: Stream write error.
SSH: SFtp, oid: 3: File handle freed.
SSH: SFtp, oid: 3: Session deinitalized.
SSH: SFtp, oid: 2: Session deinitalized.
SSH: Session, oid: 1: Successfully disconnected from the server.
SSH: Session, oid: 1: Session handles freed.
I use the SSH package all the time with no problem (I am also the developer and maintainer of it)
This error "usually" means you don't have read/write access in your current session:
Failed. Code = -31, SFTP Protocol Error
SSH2 servers can setup per-user read/write access differently, that may cause error. Or it may be a path error. I can't really tell at this point. Could you manually DLOG() the path strings in debug mode, so that we can examine them.
To verify this,
Could you try the below test server and see if it is working (you should be able to navigate the server and download files, you will not have write access)
Server: test.rebex.net
Port: 22
User: demo
Password: password
Best regards,
Oblivion
Github page: https://github.com/ismail-yilmaz
Bobcat the terminal emulator: https://github.com/ismail-yilmaz/Bobcat
[Updated on: Wed, 18 December 2024 14:17] Report message to a moderator |
|
|
|
| Re: SFTPBrowser example build failed [message #61273 is a reply to message #61272] |
Wed, 18 December 2024 15:20 |
|
Scott_Huang
Messages: 27
Registered: December 2024
|
Promising Member |
|
|
@Oblivion, thanks for your prompt answer!
I try test the demo SFTP server which you provided, it seems work well.
However, when I switch to my SFTP server, it have the same error which I describe in last post.
I am sure my SFTP user name & password is correct and have write access right.
I guess may be dir/path error, you also mention it, so I adjust connect() function to manual set basedir as below.
session.Timeout(settings.timeout * 1000);
if((connected = session.Connect(
~settings.host,
~settings.port,
~settings.user,
~settings.password
))) {
browser.Attach(new SFtp(session));
filesystem.Mount(*browser);
basedir = browser->GetDefaultDir();
basedir = "msp";//add for testing
DLOG(basedir);
dir.SetData("msp");//add for testing
DLOG(dir.GetData().ToString());
DLOG(GetWorkdir());
DLOG("Before work dir");
Workdir(basedir);
DLOG("After work dir, and before LoadDir");
LoadDir();
DLOG("After load dir");
}
else SessionError();
}
The list still empty, but I am able upload file or create sub folder without error. And other SFTP software does show the file & folder created too.
So, that means, loaddir() function have issue.
Not able successful list folder & files.
After add more DLOG, I am sure it caused by below:
if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false))
sortbyext ? SortByExt(list) : SortByName(list);
LOG:
SSH: Session, oid: 1: Using Upp's memory managers.
SSH: Session, oid: 1: Session successfully initialized.
SSH: Session, oid: 1: Compression is disabled.
SSH: Session, oid: 1: Handshake successful.
SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive]
SSH: Session, oid: 1: Client succesfully authenticated.
SSH: SFtp, oid: 2: Session successfully initialized.
SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: /
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file
SSH: SFtp, oid: 3: Session successfully initialized.
SSH: SFtp, oid: 3: File 'msp/test.txt' is successfully opened.
SSH: SFtp, oid: 3: File attributes successfully retrieved.
SSH: SFtp, oid: 3: File handle freed.
SSH: SFtp, oid: 3: Session deinitalized.
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error ==> Scott, It caused by I try create one existing temp folder.
SSH: SFtp, oid: 2: Directory 'msp/temp2' is succesfully created.
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error ==> Scott: Should be loaddir() function caused error.
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file
I try add more DLOG:
void SFtpBrowser::LoadDir()
{
if(browser->InProgress())
return;
list.Clear();
DLOG("Before SFtpFileSystemInfo sfsi(*browser);");
SFtpFileSystemInfo sfsi(*browser);
DLOG("After SFtpFileSystemInfo sfsi(*browser);");
DLOG("Before If");
DLOG(GetWorkdir());
if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false)){
DLOG("Before sort");
sortbyext ? SortByExt(list) : SortByName(list);
DLOG("After sort");
}
DLOG("After If");
DLOG("Before Summary()");
Summary();
DLOG("After Summary(), and before Sync()");
Sync();
DLOG("After Sync()");
}
LOG Details:
SSH: Session, oid: 1: Using Upp's memory managers.
SSH: Session, oid: 1: Session successfully initialized.
SSH: Session, oid: 1: Compression is disabled.
SSH: Session, oid: 1: Handshake successful.
SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive]
SSH: Session, oid: 1: Client succesfully authenticated.
SSH: SFtp, oid: 2: Session successfully initialized.
SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: / ==> Scott: Is it the root cause? I try set to known folder "MSP" instead of "/"
msp
msp
msp
Before work dir
After work dir, and before LoadDir
Before SFtpFileSystemInfo sfsi(*browser);
After SFtpFileSystemInfo sfsi(*browser);
Before If
msp ==> Scott: I try DLOG(GetWorkdir());
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file
After If
Before Summary()
After Summary(), and before Sync()
After Sync()
After load dir
Any way to fix it?
[Updated on: Wed, 18 December 2024 15:59] Report message to a moderator |
|
|
|
| Re: SFTPBrowser example build failed [message #61276 is a reply to message #61273] |
Thu, 19 December 2024 03:14 |
|
Scott_Huang
Messages: 27
Registered: December 2024
|
Promising Member |
|
|
I try debug again, so try add list files functions in LoadDir(), that manual added function able successful list files.
So, can @Oblivion or someone else help to fix issue of if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false)){
void SFtpBrowser::LoadDir()
{
if(browser->InProgress())
return;
list.Clear();
DLOG("Before SFtpFileSystemInfo sfsi(*browser);");
SFtpFileSystemInfo sfsi(*browser);
DLOG("After SFtpFileSystemInfo sfsi(*browser);");
DLOG("Start list folder");
DLOG(GetWorkdir());
SFtp::DirList ls;
// Get a remote dir listing.
if(!browser->ListDir(GetWorkdir(), ls)) {
DLOG(browser->GetErrorDesc());
return;
}
// Filter the dir list.
auto files = FilterRange(ls, [](const SFtp::DirEntry& e) { return e.IsFile() && e.GetSize() <= 65536; });
// Download the files, using worker threads in parallel.
CoFor(min(files.GetCount(), 1000), [&](int i){
DLOG(files[i]);
});
DLOG("Before If");
if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false)){
DLOG("Before sort");
sortbyext ? SortByExt(list) : SortByName(list);
DLOG("After sort");
}
DLOG("After If");
DLOG("Before Summary()");
Summary();
DLOG("After Summary(), and before Sync()");
Sync();
DLOG("After Sync()");
}
LOGs:
SSH: Session, oid: 1: Using Upp's memory managers.
SSH: Session, oid: 1: Session successfully initialized.
SSH: Session, oid: 1: Compression is disabled.
SSH: Session, oid: 1: Handshake successful.
SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive]
SSH: Session, oid: 1: Client succesfully authenticated.
SSH: SFtp, oid: 2: Session successfully initialized.
SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: /
/msp/
/msp/
/msp/
Before work dir
After work dir, and before LoadDir
Before SFtpFileSystemInfo sfsi(*browser);
After SFtpFileSystemInfo sfsi(*browser);
Start list folder
/msp/
SSH: SFtp, oid: 2: Directory '/msp/' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (4 entries)
SSH: SFtp, oid: 2: File handle freed.
-rw-r--r-- 30015 1485 411 12/19/2024 01:24:12 12/19/2024 01:24:12 MSP~some file name.csv
Before If
SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error
SSH: SFtp, oid: 2: Failed. Code = -31, Failed opening remote file
After If
Before Summary()
After Summary(), and before Sync()
After Sync()
After load dir
|
|
|
|
|
|
| Re: SFTPBrowser example build failed [message #61278 is a reply to message #61277] |
Thu, 19 December 2024 05:41 |
|
Scott_Huang
Messages: 27
Registered: December 2024
|
Promising Member |
|
|
Hello Oblivion,
I try do some minor adjustment to testing the last sample:
String GetWorkdir() const {
String workDir = dir.GetData().ToString();
if(workDir == "/") return Null;
else return workDir;
}
Which intend set default as empty instead of "/", at least it work for 1st top layer.
(Still some bug, such as it would show as folder icon if I upload a test.txt file)
But when navigate to sub folder, it will not able correct LoadDir() again.
Then I try
DLOG(GetWorkdir());
SFtp::DirList ls;
// Get a remote dir listing.
if(!browser->ListDir(GetWorkdir(), ls)) {
DLOG(browser->GetErrorDesc());
return;
}
// Filter the dir list.
//auto files = FilterRange(ls, [](const SFtp::DirEntry& e) { return e.IsFile() && e.GetSize() <= 65536; });
auto files = FilterRange(ls, [](const SFtp::DirEntry& e) { return true; });
// Download the files, using worker threads in parallel.
CoFor(min(files.GetCount(), 1000), [&](int i){
DLOG(files[i]); });
And then I try testing sfsi, since we know that portion have issue:
DLOG("Start for sfsi.Find");
Array<FileSystemInfo::FileInfo> root = sfsi.Find("*");
for(int i = 0; i < root.GetCount(); i++){
DLOG(root[i].filename);
//DLOG(root[i].root_style);
//DLOG(root[i].root_desc);
}
DLOG("End for sfsi.Find");
It is work at 1st time when GetWordir() return Null, but failed when I select sub folder from dir dropdownlist.
Note, the SFTP server itself is Posix, since sfsi.IsPosix() return true, while my working computer is Windows.
So, I think the issue may be in Load() function in FileSel.cpp since at least work when if(dir.IsEmpty()) .
Array<FileSystemInfo::FileInfo> ffi =
filesystem.Find(AppendFileName(dir, filesystem.IsWin32() ? "*.*" : "*"));
And then I try reference/SFtpFileSel example per your suggestion for testing, the result as below:
|
|
|
|
| Re: SFTPBrowser example build failed [message #61283 is a reply to message #61278] |
Thu, 19 December 2024 08:11 |
Oblivion
Messages: 1266
Registered: August 2007
|
Senior Contributor |
|
|
Hello Scott,
Thank you for your patience and the info.
Quote:Note, the SFTP server itself is Posix, since sfsi.IsPosix() return true, while my working computer is Windows.
That's because in most systems, SSH2 is configured to run using POSIX style file paths. Usually, unless you have a niche SSH server implementation, it should be ok.
E.g. I run OpenSSH on my windows machine and connect to windows machines with OpenSSH implementation, and the example works fine on them.
By the way, what is your ssh configuration? (not any personal info or paths, just the server's general config) I am going to see what is actually going on, and fix and improve the example but it would be nice to replicate a similar problem on my machines.
P.s: In my windows setups, the directory/file can be created or uploaded. However, the file list is not updated/refreshed after the operation. It is missing in the code. So that's one bug to fix 
Best regads,
Oblivion
Github page: https://github.com/ismail-yilmaz
Bobcat the terminal emulator: https://github.com/ismail-yilmaz/Bobcat
[Updated on: Thu, 19 December 2024 09:03] Report message to a moderator |
|
|
|
| Re: SFTPBrowser example build failed [message #61284 is a reply to message #61283] |
Thu, 19 December 2024 10:00 |
|
Scott_Huang
Messages: 27
Registered: December 2024
|
Promising Member |
|
|
Hello Oblivion,
Thanks for commit would make some time to fix bug. And it is not easy since not able duplicate similar env in your computer.
The SFTP server I try to connect is setup by my company IT team(A big company), I do not have enough info of it.
Should be shared to multiple people.
But does able smoothly connect and list/download/upload data through other 3rd party software, such as RealFTP, WinSCP.
I also able use PHP to manipulate that SFTP server data.
The ultimatepp SFTP function actually should be work too, such as able SFtp::DirList and so on so forth.
And it work well if connect to your demo SFTP server.
But need you help deep dive below code, such as Load() and sfsi portion, it must have something not consider well.
if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false))
sortbyext ? SortByExt(list) : SortByName(list);
[Updated on: Thu, 19 December 2024 10:08] Report message to a moderator |
|
|
|
|
|
|
|
| Re: SFTPBrowser example build failed [message #61291 is a reply to message #61290] |
Fri, 20 December 2024 03:07 |
|
Scott_Huang
Messages: 27
Registered: December 2024
|
Promising Member |
|
|
Hi Oblivion,
Yeah, it work as expected if under DEBUG build mode after update LoadDir() as below:
Able list/download/upload file without issues.
Note: As for build output mode, I select All static and Default debug info leve = None, and add below additional line:
f.hidden = false;//add this sentence, or else summary function would not count it.
void SFtpBrowser::LoadDir()
{
if(browser->InProgress())
return;
list.Clear();
SFtp::DirList ls;
if(browser->ListDir(GetWorkdir(), ls)) {
for(const SFtp::DirEntry& e : ls) {
String s = e.GetName();
if(s == "." || s == "..") continue;
FileList::File f; // Can be easily costumized...
f.name = s;
f.time = e.GetLastModified();
f.font = StdFont();
f.ink = SColorText;
f.icon = e.IsFile() ? CtrlImg::File() : CtrlImg::Dir();
f.isdir = e.IsDirectory();
f.length = e.GetSize();
f.extink = SLtBlue;
f.hidden = false;//Scott: add this sentence, or else summary function would not count it.
list.Add(f);
}
sortbyext ? SortByExt(list) : SortByName(list);
}
Summary();
Sync();
}
But, there still have bug if under Release build mode: which is not able upload file, always error and end with 0 byte.
LOG:
SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (2 entries)
SSH: SFtp, oid: 2: File handle freed.
//Scott: I add 'RLOG(Format("dest: %s, UnixPath: %s ",dest,UnixPath(dest)));' in upload function for debug since assume may be path error.
dest: //msp/config/temp\test.txt, UnixPath: //msp/config/temp/test.txt
SSH: SFtp, oid: 3: Session successfully initialized.
SSH: SFtp, oid: 3: File '//msp/config/temp/test.txt' is successfully opened.
SSH: SFtp, oid: 3: File attributes successfully retrieved.
SSH: SFtp, oid: 3: EOF received.
SSH: SFtp, oid: 3: Stream write error.
SSH: SFtp, oid: 3: File handle freed.
SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (3 entries)
SSH: SFtp, oid: 2: File handle freed.
SSH: SFtp, oid: 3: Session deinitalized.
SSH: SFtp, oid: 2: File '//msp/config/temp/test.txt' is successfully deleted.
SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (2 entries)
SSH: SFtp, oid: 2: File handle freed.
So, please help on below:
1. Why debug mode able upload file without issue while release mode encounter EOF and create 0 byte file?
2. It will be show SFTP protocal error in pop up alert if intend delete a none empty folder.
(in debug log file it show:SSH: SFtp, oid: 2: Failed. Code = -31, SFTP Protocol Error)
So, suggest show frendly alert to say the folder not empty and cannot delete, instead of execute and then failed.
3. Do you know why before old code is wrong? Please better also find out root cause of sfsi or Upp::Load function too. Thanks.
if(Upp::Load(list, GetWorkdir(), "*", false, Null, (FileSystemInfo&) sfsi, Null, false))
sortbyext ? SortByExt(list) : SortByName(list);
[Updated on: Fri, 20 December 2024 04:18] Report message to a moderator |
|
|
|
| Re: SFTPBrowser example build failed [message #61292 is a reply to message #61291] |
Fri, 20 December 2024 06:05 |
Oblivion
Messages: 1266
Registered: August 2007
|
Senior Contributor |
|
|
Hello Scott,
In order to see what's wrong with writing a file and isolate it (Unfortunately I can't reproduce the error), could you replace the below method. Use a small file to test uploading please.
void SFtpBrowser::Transfer(int opcode, const String src, const String& dest)
{
SFtp worker(session);
Progress pi(this, src);
worker.WhenProgress = [&pi](int64 done, int64 total)
{
pi.SetText(
Format(
t_("%1:s of %2:s is transferred"),
FormatFileSize(done),
FormatFileSize(total)
)
);
return pi.SetCanceled(int(done), int(total));
};
pi.Create();
switch(opcode) {
case GET: {
pi.Title(t_("Downloading ") << GetFileName(src));
FileOut fout(dest);
if(fout && !worker.LoadFile(fout, src))
ErrorOK(DeQtf(worker.GetErrorDesc()));
break;
}
case PUT: {
pi.Title(t_("Uploading ") << GetFileName(src)); // Not for use, only for testing
String s = LoadFile(src);
if(auto h = worker.OpenWrite(dest); h) {
int n = worker.Put(h, ~s, s.GetLength());
RLOG(String() << "Uploaded: " << n << ", filesize: " << s.GetLength());
RDUMP(worker.IsError());
if(worker.IsError())
ErrorOK(DeQtf(worker.GetErrorDesc()));
worker.Close(h);
}
else
RLOG("Couldn't open file: " << src << ", error: " << worker.GetErrorDesc());
break;
}
}
LoadDir();
}
Best regards,
Oblivion
Github page: https://github.com/ismail-yilmaz
Bobcat the terminal emulator: https://github.com/ismail-yilmaz/Bobcat
|
|
|
|
| Re: SFTPBrowser example build failed [message #61294 is a reply to message #61292] |
Fri, 20 December 2024 08:25 |
|
Scott_Huang
Messages: 27
Registered: December 2024
|
Promising Member |
|
|
Hi Oblivion,
It worked without error in Release build mode with below new code:
This new method try write src into a String , and then write String to dest, so no issue of small/middle files,
but would losing visibility of progressing % during upload. Anyway to load by chunk and show progressing under new code? Thanks.
case PUT: {
pi.Title(t_("Uploading ") << GetFileName(src)); // Not for use, only for testing
String s = LoadFile(src);
if(auto h = worker.OpenWrite(dest); h) {
int n = worker.Put(h, ~s, s.GetLength());
RLOG(String() << "Uploaded: " << n << ", filesize: " << s.GetLength());
RDUMP(worker.IsError());
if(worker.IsError())
ErrorOK(DeQtf(worker.GetErrorDesc()));
worker.Close(h);
}
else
RLOG("Couldn't open file: " << src << ", error: " << worker.GetErrorDesc());
break;
}
old wrong code is:
case PUT: {
pi.Title(t_("Uploading ") << GetFileName(src));
FileIn fin(src);
if(fin && !worker.SaveFile(dest, fin))
ErrorOK(DeQtf(worker.GetErrorDesc()));
break;
}
Note: The most tricky thing is old code failed upload file when build by CLANG RELEASE & DEBUG MODE or VS2022 RELEASE MODE. Only work in VS2022 DEBUG build mode.
Do you able point out the key gap of the old code and try fix it? It is better have upload progressing bar too:) Thanks.
Or else, I have to use VS2022 DEBUG BUILD,then exe size is big. Or use your new code under RELEASE BUILD, but losing upload progressing progress.
Thanks.
Regards,
Scott Huang
[Updated on: Fri, 20 December 2024 08:41] Report message to a moderator |
|
|
|
|
|
| Re: SFTPBrowser example build failed [message #61296 is a reply to message #61295] |
Fri, 20 December 2024 09:20 |
|
Scott_Huang
Messages: 27
Registered: December 2024
|
Promising Member |
|
|
Hello Oblivion,
Good to know you will enhance code again to add progressing %. Thnaks.
In the meanwhile, I also try upgrade the old code and testing by myself with below update:
case PUT: {
pi.Title(t_("Uploading ") << GetFileName(src));
FileIn fin(src);
/*
if(fin && !worker.SaveFile(dest, fin))
ErrorOK(DeQtf(worker.GetErrorDesc()));
break;
*/
SFtpFileOut fout(worker, dest);
int64 size = fin.GetSize(), count = 0;
int64 chunkSize = 1024 * 64;
Buffer<byte> chunk(chunkSize, 0);
Gate<int64, int64> WhenProgress;
WhenProgress(0, size);
while(!fin.IsEof()) {
RLOG(Format("size %d, count %d, chunkSize %d, min %d", size, count, chunkSize, (int)min<int64>(size - count, chunkSize) ));
int n = fin.Get(chunk, (int) min<int64>(size - count, chunkSize));
RLOG(Format("n %d",n));
if(n > 0) {
fout.Put(chunk, n);
if(fout.IsError()) {
RLOG("Stream write error. " + fin.GetErrorText() + " out stream: "+fout.GetErrorText());
ErrorOK(DeQtf(worker.GetErrorDesc())+"Stream write error. " + fin.GetErrorText() + " out stream: "+fout.GetErrorText());
break;
}
count += n;
if(WhenProgress(count, size)) {
break;
}
}
if(fin.IsError()) {
RLOG("Stream read error. " + fin.GetErrorText());
ErrorOK(DeQtf(worker.GetErrorDesc())+"Stream read error. " + fin.GetErrorText());
break;
}
}
}
The LOG:
SSH: SFtp, oid: 2: File handle freed.
dest: /\test.txt, UnixPath: //test.txt
SSH: SFtp, oid: 3: Session successfully initialized.
SSH: SFtp, oid: 3: File '//test.txt' is successfully opened.
SSH: SFtp, oid: 3: File attributes successfully retrieved.
size 47, count 0, chunkSize 65536, min 47
n 47
SSH: SFtp, oid: 3: EOF received.
Stream write error. out stream: A non-blocking socket operation could not be completed immediately.
SSH: SFtp, oid: 3: File handle freed.
SSH: SFtp, oid: 2: Directory '/' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (7 entries)
SSH: SFtp, oid: 2: File handle freed.
SSH: SFtp, oid: 3: Session deinitalized.
May be you can checking why EOF received? Is it caused by non-blocking socket & blocking socket different if compare with your SFTP and my company's SFTP server?
Thanks.
Regards,
Scott Huang
[Updated on: Fri, 20 December 2024 09:25] Report message to a moderator |
|
|
|
|
|
| Re: SFTPBrowser example build failed [message #61299 is a reply to message #61297] |
Fri, 20 December 2024 15:42 |
|
Scott_Huang
Messages: 27
Registered: December 2024
|
Promising Member |
|
|
Hi Oblivion,
Merry Christmas!
Actually I testing different size files before last post, including 1M, all failed and create 0 size file in SFTP server.
SFtpFileOut should have some bug in case some SFTP server not similar as your demo SFTP server.
Anyway, I have upgrade code with your hint as below, it can successful upload files and also show progress %:
void SFtpBrowser::Transfer(int opcode, const String src, const String& dest)
{
SFtp worker(session);
Progress pi(this, src);
worker.WhenProgress = [&pi](int64 done, int64 total) {
pi.SetText(Format(t_("%1:s of %2:s is transferred"), FormatFileSize(done),
FormatFileSize(total)));
return pi.SetCanceled(int(done), int(total));
};
pi.Create();
switch(opcode) {
case GET: {
pi.Title(t_("Downloading ") << GetFileName(src));
FileOut fout(dest);
if(fout && !worker.LoadFile(fout, src))
ErrorOK(DeQtf(worker.GetErrorDesc()));
break;
}
case PUT: {
pi.Title(t_("Uploading ") << GetFileName(src)); // Not for use, only for testing
String s = LoadFile(src);
if(auto h = worker.OpenWrite(dest); h) {
/* int n = worker.Put(h, ~s, s.GetLength());
RLOG(String() << "Uploaded: " << n << ", filesize: " << s.GetLength());
RDUMP(worker.IsError());
if(worker.IsError())
ErrorOK(DeQtf(worker.GetErrorDesc()));
worker.Close(h);*/
int64 size = s.GetLength(), count = 0;
if(settings.log)
RLOG(Format("File size %d", size));
int chunkSize = 1024 * 64, thisTimeSize = 0;
worker.WhenProgress(0, size);
while(!(count >= size)) {
thisTimeSize = (int)min<int64>(size - count, chunkSize);
if(settings.log)
RLOG(Format("======= size %d, count %d, chunkSize %d, min %d", size, count,
chunkSize, thisTimeSize));
int x = worker.Put(h, (~s) + count, thisTimeSize);
count += x;
if(settings.log)
RLOG(String() << "This time should be uploaded: " << thisTimeSize
<< ", Real uploaded: " << x << ", Current Accumulated Count:"
<< count << ", total filesize: " << size);
// RDUMP(worker.IsError());
if(worker.WhenProgress(count, size)) {
break;
}
if(worker.IsError()) {
ErrorOK("Upload failed : " + DeQtf(worker.GetErrorDesc()));
break;
}
}
worker.Close(h);
break;
}
else if(settings.log)
RLOG("Couldn't open file: " << src << ", error: " << worker.GetErrorDesc());
break;
}
}
LoadDir();
}
You can consider this issue fixed.
But could you help improve it when you have time:
1. Any better way if compare to above? (I am new for UPP)
2. Enhance it to real multiple threads, above actually is single upload thread, quite slow if compare to 3rd party SFTP software.
3. Why stream way failed and we have switch to one time read file into String and then chunk write into SFTP?
Regards,
Scott Huang
[Updated on: Sat, 21 December 2024 04:44] Report message to a moderator |
|
|
|
| Re: SFTPBrowser example build failed [message #61301 is a reply to message #61299] |
Sat, 21 December 2024 11:11 |
Oblivion
Messages: 1266
Registered: August 2007
|
Senior Contributor |
|
|
Hello Scott,
Thank you for your patience and feedback.
I see no difference between your updated transfer code and SFtp::CopyData, which is used by SFtpFileStream derived class.
However, I suspect the server's filesystem does not support truncation, which is set by SFtpFileOut, by default, and it is NOT set by plain SFtp::OpenWrite method.
That might be the culprit, newer servers, especially flash-based ones don't support it, so the operation fails. (At least my theory).
Can you modify the SFtpStream.cpp file in SSH package and then test the original code with streams?
bool SFtpStream::Open(SFtp& sftp_, const char *filename, dword mode, int acm)
{
if(IsOpen())
Close();
sftp = &sftp_;
int iomode = mode & ~SHAREMASK;
handle = sftp->Open(filename,
iomode == READ
? SFtp::READ
: iomode == CREATE
? SFtp::READ|SFtp::WRITE|SFtp::CREATE|SFtp::TRUNCATE // <- remove SFtp::TRUNCATE
: SFtp::READ|SFtp::WRITE,
acm
);
if(handle) {
SFtpAttrs attrs;
if(!sftp->GetAttrs(handle, attrs)) {
sftp->Close(handle);
handle = nullptr;
return false;
}
OpenInit(mode, attrs.filesize);
}
return handle;
}
I will try to answer your other questions as well.
Best regards,
Oblivion
Github page: https://github.com/ismail-yilmaz
Bobcat the terminal emulator: https://github.com/ismail-yilmaz/Bobcat
[Updated on: Sat, 21 December 2024 11:12] Report message to a moderator |
|
|
|
| Re: SFTPBrowser example build failed [message #61304 is a reply to message #61301] |
Sun, 22 December 2024 02:30 |
|
Scott_Huang
Messages: 27
Registered: December 2024
|
Promising Member |
|
|
Hi Oblivion,
I try remove SFtp::TRUNCATE as you said, it did not work, same error.
I also testing remove both SFtp::CREATE|SFtp::TRUNCATE from SFtpStream.cpp, then the behavior like below:
1. Failed if SFTP file not existing, it show error: Failed opening remote file. I think it caused by I remove SFtp::CREATE flag too.
2. I try use other SFTP software to upload same file firstly. And then U++ exe can upload file, but it would write content to existing file, double contents.
There should be have other subtle gaps if compare to OpenWrite function:
Open(path, CREATE | WRITE, IRALL | IWUSR);
I personal guess it caused by below code, so add several RLOG for debug: (test.txt file size is 47 byte)
if(handle) {
RLOG("Have handle, before checking GetAttrs");
SFtpAttrs attrs;
if(!sftp->GetAttrs(handle, attrs)) {
RLOG("Failed get attrs");
sftp->Close(handle);
handle = nullptr;
return false;
}
else{
RLOG("Succes get attrs");
RLOG("===Show filesize");
DUMP(attrs);
RLOG(attrs.filesize);
RLOG((int64)attrs.filesize);
RLOG("----");
}
OpenInit(mode, attrs.filesize);
}
void BlockStream::OpenInit(dword mode, int64 _filesize) {
streamsize = _filesize;
RLOG(String()<<"++In OpenInit function - DEBUG: file size"<<_filesize);
RLOG(_filesize);
RLOG(streamsize);
RLOG("---------------");
LOG does show attributes retrievedm, so the issue should be in function OpenInit(mode, attrs.filesize); And why additional session created
SSH: Session, oid: 1: Starting DNS sequence locally for deft.dell.com:22
SSH: Session, oid: 1: Successfully connected to deft.dell.com:22
SSH: Session, oid: 1: Using Upp's memory managers.
SSH: Session, oid: 1: Session successfully initialized.
SSH: Session, oid: 1: Compression is disabled.
SSH: Session, oid: 1: Handshake successful.
SSH: Session, oid: 1: Authentication methods list successfully retrieved: [publickey,password,keyboard-interactive]
SSH: Session, oid: 1: Client succesfully authenticated.
SSH: SFtp, oid: 2: Session successfully initialized.
SSH: SFtp, oid: 2: Symbolic link operation is successful. Target: /
SSH: SFtp, oid: 2: Directory '/' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (6 entries)
SSH: SFtp, oid: 2: File handle freed.
SSH: SFtp, oid: 2: Directory '//msp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (3 entries)
SSH: SFtp, oid: 2: File handle freed.
SSH: SFtp, oid: 2: Directory '//msp/config' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (171 entries)
SSH: SFtp, oid: 2: File handle freed.
SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (2 entries)
SSH: SFtp, oid: 2: File handle freed.
dest: //msp/config/temp\test.txt, UnixPath: //msp/config/temp/test.txt
++In OpenInit function - DEBUG: file size47
47
47
---------------
SSH: SFtp, oid: 3: Session successfully initialized.
SSH: SFtp, oid: 3: File '//msp/config/temp/test.txt' is successfully opened.
Have handle, before checking GetAttrs
SSH: SFtp, oid: 3: File attributes successfully retrieved.
Succes get attrs
===Show filesize
839918903936
839918903936
----
++In OpenInit function - DEBUG: file size839918903936
839918903936
839918903936
---------------
DEBUG: NOT READ
SSH: SFtp, oid: 3: EOF received.
SSH: SFtp, oid: 3: Stream write error.
SSH: SFtp, oid: 3: File handle freed.
SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (3 entries)
SSH: SFtp, oid: 2: File handle freed.
SSH: SFtp, oid: 3: Session deinitalized.
SSH: SFtp, oid: 2: File '//msp/config/temp/test.txt' is successfully deleted.
SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (2 entries)
SSH: SFtp, oid: 2: File handle freed.
Questions: Why addition SSH oid created? Why file size wrong?
Anyway, I am ok with my last post to use worker.OpenWrite & worker.Put, it worked.
Attach good code here for others reference.
Regards,
Scott Huang
[Updated on: Sun, 22 December 2024 07:12] Report message to a moderator |
|
|
|
| Re: SFTPBrowser example build failed [message #61306 is a reply to message #61304] |
Sun, 22 December 2024 11:41 |
Oblivion
Messages: 1266
Registered: August 2007
|
Senior Contributor |
|
|
Hello Scott,
Thank you for your help. It would be better for us all if we fix the issue instead of using a workaround, if possible.
Given your latest log, it appears that the structure returns an invalid file size for the newly created file (it should be 0).
Hopefully, the below patch fixes the issue once and for all:
bool SFtpStream::Open(SFtp& sftp_, const char *filename, dword mode, int acm)
{
if(IsOpen())
Close();
sftp = &sftp_;
int iomode = mode & ~SHAREMASK;
handle = sftp->Open(filename,
iomode == READ
? SFtp::READ
: iomode == CREATE
? SFtp::READ|SFtp::WRITE|SFtp::CREATE|SFtp::TRUNCATE
: SFtp::READ|SFtp::WRITE,
acm
);
if(handle) {
SFtpAttrs attrs;
Zero(attrs);
if(iomode != CREATE && !sftp->GetAttrs(handle, attrs)) {
sftp->Close(handle);
handle = nullptr;
return false;
}
OpenInit(mode, attrs.filesize); // filesize always should be 0 for create|truncate
}
return handle;
}
I would be grateful if you could try this patch and report back.
Also, I have updated the example (attached). It can now:
1) set chunksize
2) set waitstep
3) set compression
4) allows dropping files onto browser for uploading.
As for your other questions:
1. Please keep in mind that SFtpBrowser is a very simple example, demonstrating how to integrate SSH package into a GUI. It is not intended to be a rival for WinSCP or any other commercial app.
Of course, SSH package allows you to write such applications. In fact I do have a such professional app and its performance is on par with WinScp, if not faster.
To achieve faster transfer speeds with SFtpFileBrowser, you can enable compression and increase the chunksize (preferably to 1 Mib).
2. Again, this is a simple example. However, file partitioning (parallel download of a single, huge file) is possible with SSH package, but it would be a complex app, beyond the scope of this demo. (my app does that too).
Best regards,
Oblivion
Github page: https://github.com/ismail-yilmaz
Bobcat the terminal emulator: https://github.com/ismail-yilmaz/Bobcat
[Updated on: Sun, 22 December 2024 12:12] Report message to a moderator |
|
|
|
|
|
|
|
| Re: [RESOLVED] Re: SFTPBrowser example build failed [message #61309 is a reply to message #61308] |
Sun, 22 December 2024 15:54 |
|
Scott_Huang
Messages: 27
Registered: December 2024
|
Promising Member |
|
|
Hello Oblivion,
Per testing, your new code also working for Upload. That SFTP server seems support SCP transfer for upload. Thanks.
Btw, do SCP mean Secure Copy Protocal, why it would be faster than normal SFTP way? thanks.
But SCP way failed for download. (SFTP way good for download if I change setting back)
So, split SCP for upload only? and download still use SFTP way, but enhanced to multiple thread when you have time? Thanks.
LOG:
SSH: Scp, oid: 10: Unable to open file //msp/config/temp/test.txt
SSH: Scp, oid: 10: Failed. Code = -28, Failed to recv file
SSH: SFtp, oid: 2: Directory '//msp/config/temp' is successfully opened.
SSH: SFtp, oid: 2: Directory listing is successful. (5 entries)
SSH: SFtp, oid: 2: File handle freed.
Appreciate your continious enhancements!
Take your time of below. Merry Christmas!
Quote:
Then, I'll write a simple example for parallel download (file partitioning). But it will take some time, probably within this week.
Regards,
Scott Huang
[Updated on: Mon, 23 December 2024 03:48] Report message to a moderator |
|
|
|
| Re: [RESOLVED] Re: SFTPBrowser example build failed [message #61314 is a reply to message #61309] |
Mon, 23 December 2024 08:50 |
Oblivion
Messages: 1266
Registered: August 2007
|
Senior Contributor |
|
|
Hello Scott,
Quote:Btw, do SCP mean Secure Copy Protocal, why it would be faster than normal SFTP way?
Well, long story short: SCP is different from SFTP in that SFTP has a protocol overhead (it internally chunks and "acks" transfered chunks), so it will be slower on big file transfers.
The downside of scp is that it has no transfer/pause/continue mechanism, as it is only a data stream over SSH protocol. Difference is usually visible in huge file transfers.
SCP download error: Possibly your server not supporting downloads, and only uploads by SCP?
As for the file partitioning: I decided to integrate a dumbed-down version of it (can be selected in the settings) into the SFtpBrowser within a couple of days.
Best regards,
Oblivion
Github page: https://github.com/ismail-yilmaz
Bobcat the terminal emulator: https://github.com/ismail-yilmaz/Bobcat
[Updated on: Mon, 23 December 2024 08:51] Report message to a moderator |
|
|
|
|
|
| Re: [RESOLVED] Re: SFTPBrowser example build failed [message #61343 is a reply to message #61315] |
Sat, 28 December 2024 11:18 |
Oblivion
Messages: 1266
Registered: August 2007
|
Senior Contributor |
|
|
Hello Scott,
I have attached the latest SFtpBrowser example (with parallel download, namely file partitioning).
As I have noted in the code, ideally we would create cosessions only once (on login) and then use them as needed.
But that would make the example very complex. In order to reduce the complexity in this example I am using a more direct approach.
Keep in mind that it would make no sense to use SFTP parallel download feature with small files. It is useful for downloading large files (yet, in most cases a single threaded SCP tranfer will still be faster)
Also, I omitted some checks that would be better implemented in real-life situations.
It is up to you to write a better SFtpBrowser. In fact I encourage you to do so. This is what example codes are for.
IIRC, this is the first time I am demonstrating the parallel download feature.
Best regards,
Oblivion
Github page: https://github.com/ismail-yilmaz
Bobcat the terminal emulator: https://github.com/ismail-yilmaz/Bobcat
[Updated on: Sat, 28 December 2024 11:23] Report message to a moderator |
|
|
|
| Re: [RESOLVED] Re: SFTPBrowser example build failed [message #61344 is a reply to message #61343] |
Sat, 28 December 2024 16:29 |
|
Scott_Huang
Messages: 27
Registered: December 2024
|
Promising Member |
|
|
Hello Oblivion,
Appreciate you share the tips of filepart/LoadFilePart/CoLoadFile, really helpful.
It works only except one bug, we need add 'connected = true' in Connect function after successful connect to SFTP, or else, the mkdir & upload & context would not enabled. Thanks.
if(CreateSession(session)) {
connected = true;//Add by Scott
browser.Attach(new SFtp(session));
basedir = browser->GetDefaultDir();
Workdir(basedir);
LoadDir();
}
else
SessionError();
Regards,
Scott Huang
[Updated on: Sat, 28 December 2024 17:13] Report message to a moderator |
|
|
|
Goto Forum:
Current Time: Sun May 10 00:06:15 GMT+2 2026
Total time taken to generate the page: 0.01299 seconds
|
U++ framework
Search
Help
Members
Register
Login
Home
