In this post of oracle service cloud integration let’s discuss key design options based on your file size, content format and delivery (outbound) particularly files above 10MB.
The integration cloud offers several designs and modeling options to process large files. The Download file option allows the loading of files up to 1GB in Integration Cloud local file system.
The file can be received in OIC from File Transfer Protocol (sFTP) server or through REST/SOAP APIs. This file then read in segments using the stage activity.
If you are new to Oracle Integration Cloud (OIC), then check out our blogs below:
- Oracle Integration Cloud (OIC) For Beginners Overview
- Create Oracle Integration Cloud Instance: Step-by-Step
- Console Walkthrough of Oracle Integration Cloud (OIC) Service
Handling Files less Than 1MB
If your File size is less than 1 MB, you can use Any Adapters for Inbound Interfaces with no limitation for Runtime Considerations.
Handling Files Greater Than 1MB But Less Than 10MB
File greater than 1MB but less than 10MB, you can use Any Adapters for Inbound Interfaces with some limitations at Runtime.
File Processing Limitations: Only XML or CSV format supported, Read Entire File in one go is not supported so you need to use Read File in Segments in Stage File Action.
For Outbound, you can use any Adapters that support both BASE64 encoding and file attachment.
Handling Files Greater Than 10MB
For Files greater than 10MB, we have to use FTP, REST and SOAP adapters, with some limitations at Runtime.
File Processing Limitations: Only XML or CSV format supported, Read Entire File in one go is not supported so you need to use Read File in Segments in Stage File Action.
For Outbound, you can use only use adapter that supports file attachment like FTP, REST and SOAP.
Check Out : New Homepage Of Oracle Integration Cloud.
Large File Handling
Integration Cloud does not allow loading files larger than 10MB for performance reasons.
If you need to read larger files, there is an option to “Download file”. This option handles files up to the size of 1 GB and allows files to be directly downloaded into OIC local file system without reading into memory.
The Downloaded files are accessed with the help of Stage Activity. It provides simple access to the Integration Cloud local file system.
Stage Activity provides options to list, read, write, zip and unzip files from logical directory paths used to download the file in the underlying file system.
Read More : About Invoking One Integration From Another
There is also a specialized operation for reading files in segments which is useful when files are larger (up to 1 GB).
Stage File Action ‘Read file in Segments’ allows us to specify the segment size, in numbers.
This number specifies the parts of the file per reading cycle. The stage activity along with a sequential process provides a parallel processing option that further enhanced performance.
File Adapter
The File adapter allows integrating with remote file servers for reading and writing files. You can use the File adapter with connectivity agent in scenarios where it is not possible to open up the FTP server to the Integration cloud.
The remote file is read using the File adapter trigger. Writing to remote files can be configured using the file adapter as invoke.
FTP Adapter
The FTP Adapter enables the integration of FTP servers (FTP/sFTP protocols) to integration cloud. It does not require a connectivity agent and the remote FTP server should be made accessible from the Integration cloud.
FTP Adapter cannot be used as a trigger but you can use Scheduled orchestration and add FTP as invoke for modeling trigger type integrations.
SOAP Adapter
Service can be passed as payload to an external SOAP endpoint by the SOAP Adapter. Any response received from the endpoint can be sent to the next action in the integration for further processing.
In the context of file handling, the SOAP adapter supports Message Transmission Optimization Mechanism (MTOM) attachments and inline attachments.
MTOM and inline attachments are often used together with File contents in integration flows. SOAP adapter allows inline attachments for files up to 10MB and MTOM attachments for files upto 1GB.
REST Adapter
Representational State Transfer (REST) adapter enables you to exchange messages between remote clients or web service servers and the Oracle Integration Cloud Service(OIC). It supports dynamic URLs, REST API polling, multiple operations per channel as well as XML and JSON data formats.
REST adapter provides supports for multipart message attachments and unstructured payloads for both inbound and outbound directions for files up to 1GB. The attachment contents are usually used along with file processing requirements in integration flows.
Conclusion
In this post, we have discussed how to process large files in OIC, Download file in OIC local file system and reading file in segments.
We also discussed the Adapter that will be used for Achieving this and what they offer.
If you find this post useful and have further related questions then comment below.
Related Links/References:
- 1z0-1042-20
- Oracle Integration Cloud (OIC) Instance Edition, Message Pricing, and License Types
- Oracle Cloud Documentation Links: OIC | SOA-CS | OCI
- SOAP Adapter Capabilities
- REST Adapter Capabilities
- Oracle Documentation: Clone an Integration
- How to exporting Business Data and Visual Builder Cloud Service (VBCS)
- How Import Business Data in Visual Builder Cloud Service (VBCS)
- Oracle Integration: ICS to OIC Migration
Next Task For You
Do you want to learn more about Oracle and confuse where to start or which certification is right for you? Then, click on the register now button below to register for a Free Masterclass on Oracle [1Z0-1042] Certified Cloud Integration Expert, which will help you better understand and choose the right path and clear the certification exam.
Shubhsis says
How can i change the Segment size from 200 to any other number? Is it possible? As it is disabled for me and i can not change the default 200 value.
Harshit Yadav says
Hi Shubhsis,
In recent update, oracle has remove the option to select segment size now the size is selected by OIC itself.
Some instance on old version still have this option.
Regards
Harshit Yadav
K21Academy
Sumit Kumar says
Hi Harshit,
Thanks for putting up this nice informative document!
This would really help many developers in deciding the approach for integration solution.
We have a use case where we are doing export bulk data and in callback We are getting the zip file reference along with its UCM document id.
We need to unzip the file and take out csv output file, this csv file we need to read in OIC and write to an on premise DB table.
We are using SOAP adapter to get the zip file in base64 text using “getDocumentForDocumentId” of ErpIntegrationService. We are not using MTOM attachment.
We were able to fetch the file size of more than 100 mb earlier and were able to write data with more than 5 lac records.
But now for some cases the SOAP call is failing to provide the response, saying the payload size is more than the threshold value(10 mb).
We did observe that it is still working perfectly for some files with size 100 plus mbs.
We are wondering why it is failing to fetch the file with 80mb and not failing for 101mb file (as Base64 text) without attachment.
As an alternative, we are planning to use the SOAP adapter attachment option or the file reference which we get in callback using ERP adapter.
We are not sure how we can use reference to write to OIC stage file.
We can write the reference directly to SFTP with No Schema but for stage operation we need to provide schema compulsorily. Since it is a zip file which we want to write to stage we are not sure what schema we should provide.
For the previous approach where we were fetching Base64 text as content in response payload, we were using Opaque schema to write it to stage and then read with BI report csv schema and finally write to DB table, it was/is working perfectly.
We would appreciate your valuable inputs on how to use the SOAP adapter attachment option or the file reference which we get in callback response in OIC stage operation for a zip file.
Thanks in advance!
Regards,
Sumit
sumit kumar says
Complete step for our previous approach is-:
For the previous approach where we were fetching Base64 text as content in response payload, we were using Opaque schema to write it to stage , unzipping it, listing the files for csv output file and then reading with BI report csv schema and finally writing to DB table, it was/is working perfectly.
sumit kumar says
Hi Harshit,
I was able to get different solutions for my problem .I thought of sharing it here, it might help others.
————— 1. Soap Adapter Attachment Option———–
• Invoke UCM SOAP Service
• Select Attachment Option in SOAP Adapter
• StageFile – Unzip
• StageFile – ListFiles to get the csv output file
• StageFile – Read File in Segments
• Write to DB table
Note: No need to write the zip file to stage, we can directly unzip the file from the response of SOAP call(Content field)
*******************************************************************************************
————— 2. ERP Adpter FileReference Option from Callback Message———–
• Configure callback integration for export bulk data for large volume BIP report
• In the onJobCompletion payload we get file reference of the zip file
• StageFile – Unzip with reference as FileReference which we get in callback
• StageFile – Unzip
• StageFile – ListFiles to get the csv output file
• StageFile – Read File in Segments
• Write to DB table
Note: Here also No need to write the zip file to stage, we can directly unzip the file from the callback message(FileReference field)
*******************************************************************************************
————— 3. Get CSV file as attachment instead of zip———–
• Configure callback integration for export bulk data for large volume BIP report
• From the onJobCompletion payload take the request id of report ess job
• Whenever we run export bulk data, it uploads the CSV output file separately as well to UCM
• The csv output file name in UCM tables would be like “ESS_O_219941_BIP.CSV” where 219941 is the request id of report ess job.
• Create a BIP report based on UCM tables “fusion_ocserver11g.revisions and fusion_ocserver11g.documents” , pass request id as parameter to match the docname in UCM table.
• Get the DocID of the CSV file using the BIP report mentioned above.
• Invoke UCM SOAP Service
• Select Attachment Option in SOAP Adapter
• StageFile – Read File in Segments
• Write to DB table
Note: Here we don’t need do unzip and list files stage operations as we are getting the exact CSV output file in attachment which we can directly read in segments.
This approach can be used when we have very large volume of data and if zip file increases the threshold value. Zip file contains LOG,XML and CSV files.
So getting CSV only might work in that case.
Madhu says
Thanks Sumit, your solution helps others to plan their design.
Madhu says
Great post Harshit , really useful one ????????
Rahul Dangayach says
Hi Madhu,
Glad you liked our blog.
Stay tuned for more informative blog.
Thanks and Regards
Rahul Dangayach
Team K21 Academy
sudheer says
hi,
we are extracting the data from fusion using BIP report as a CSV file. but when we extract the large data getting an error maximum allowed threshold of 10485760 bytes (10 MB).
we are using the soap adapter to call the BIP report.
Rahul Dangayach says
Hi Sudheer,
When the message payload in an integration exceeds 10 MB, you get this error.
Hope this helps.
Thanks and Regards
Rahul Dangayach
Team K21 Academy
Manjusha says
Hi,
while importing data through FBDI process , if some record is failed to import due to some reason, what will happen? should we import the hole zip file again or we can import only the failed record? How can we deal this in case of both FBDI and stage read approaches? please reply.
Thanks
Manjusha