Massive data imports via FTP to Scene7
Scene7 best practices: Massive data imports via FTP to Scene7
Prepare your source files according to the Help documentation:
Pay close attention to folder organization and structure.
Open a ticket to notify Scene7 Technical Support in advance of starting the upload job.
Provide details on number of files, volume, and approximate start date and time. This infomration notifies Scene7 Engineering and Operations of temporary extra server load, and for extra monitoring to ensure the smooth running of your upload.
Uploading to FTP
Copy the files using an ftp client to your s7ftpX.scene7.com account. Up to ten connections can be made to this account per IP address.
The FTP account is a temporary storage location for SPS ingestion. Do not use it to store files permanently.
Importing to SPS
While the files are uploading, it's unnecessary to wait for the process to complete. You can start importing right away. SPS makes a snapshot of the FTP server at the time the job starts. Any files in transit/incomplete aren't considered.
First, check your upload options:
If it's necessary to automate set creation, do so now:
Try a test upload from your desktop with the upload settings. Make sure that your images and other content look OK in SPS, and when served via the Image Servers / Viewers. Pay particular attention to set creation (if used), color profile differences, and auto cropping.
Note: Once the upload job processes the files, they are removed from the FTP.
Once you are satisfied, start an upload via FTP job considering the following:
Check the file count of the folder you want to import. Ideally, it should be around 50000 assets. While you can import more, 50000 assets is a good compromise between job duration and stability.
Separate image processing from video processing if using video transcoding. Video transcoding is server intensive, it is best to separate it from your normal image uploads initially.
It is possible to schedule the import jobs to run on a recurring method, to run once at a certain point in the future. It could allow you more flexibility but bear in mind the information above (for example, set a job to import from a subfolder to limit the volume of assets kicked off). Make sure to check the job notification emails.
Scene7 uses shared resources, so it is difficult to give a guaranteed throughput speed. Contact your consultant or Tech Support and they can give you an estimate.
The physical size of the images does not affect the speed of processing
Job log errors
You could see errors in the SPS job logs such as “Unable to process file, unsupported file format”. It usually means the file itself is corrupt, or there were packets dropped when transferring from desktop to FTP. Check that the file can be opened on the desktop (for example, in Photoshop) and if it is OK, upload it again.
Make sure that you keep local copies of your content. If you made an error, such as an incorrect upload setting, or you want to change the folder structure in SPS, it is usually easier to reupload the content into SPS, rather than moving or adjusting content in SPS. Make sure the overwrite settings are configured correctly.