Saving Daily Data in CSV Using Batch Process
Overview
Batch Process is used for time-consuming or periodically executed processes. This time, we will implement a process that saves content in CSV daily.
Before you start
In this tutorial, CSV will be saved under the following conditions:
- Batch Process will run every day at 00:00.
- It will acquire contents that belong to
topics_group_id=5and were updated the previous day. - The content to be saved includes the content ID, title, body, extended item 1, and last updated date.
- The acquired content will create a file every day and save it in
/files/ltd/topics_log/topics_yyyy-mm-dd.csv.
Creating Batch Process
Let's create a batch process.
1. Display the Batch Process List Screen
Click [Operations] -> [Batch Process] on the menu to display the batch list screen.
2. Display the Batch Process Edit Screen
Click [Add] in the upper right corner of the Batch Process list screen.
The batch editor screen will be displayed.
3. Enter the Title and Identifier
First, enter the title and identifier. Enter the following:
- Title: topics daily export
- Identifier: sample1_export_topics
- Batch: Every day, 00:00
- Member ID: Not specified
Titles and identifiers cannot be duplicated with other batch processes.
4. Describe the Content Saving Process
Next, describe the actual process. Describe the process of adding data to the editor.
Output the header
First, output the header row of the CSV to a temporary file.
Enter the following in the editor:
Replace the topics_group_id value=5 in the second line with the ID of the content definition that exists on your site.
{assign var=empty_array value='[]'|json_decode:1}
{assign var=topics_group_id value=5}
{* Execute as admin *}
{login member_id=1}
{* Get topics group *}
{capture name=method_params}{ldelim}
"topics_group_id": {$topics_group_id},
"ext_config_flg": true
{rdelim}
{/capture}
{assign var=method_params value=$smarty.capture.method_params|json_decode:1}
{api_method
var=topics_group
model="TopicsGroup"
method="details"
version="1"
method_params=$method_params
}
{* Create header line *}
{assign var=header_columns value=$empty_array}
{append var=header_columns index=topics_id value="Topic ID"}
{append var=header_columns index=subject value="Title"}
{append var=header_columns index=contents value="Content"}
{* Additional field *}
{foreach from=$topics_group.ext_config item=config}
{if $config.ext_col_nm eq 'ext_1'}
{append var=header_columns index=$config.ext_col_nm value=$config.title}
{/if}
{/foreach}
{append var=header_columns index=update_ymdhi value="Last update"}
{write_file var=path value=$header_columns}
{write_file path=$path value="\n" is_append=1}
Describe the process of acquiring content
Next, acquire the content and write it immediately after the header row.
Enter the following in the editor:
{* Fetch contents updated one day before *}
{assign var=from_date value="-2 day"|strtotime|date_format:"%Y-%m-%d"}
{assign var=to_date value="-1 day"|strtotime|date_format:"%Y-%m-%d"}
{capture name=method_params}{ldelim}
"topics_group_id": [{$topics_group_id}],
"cnt": 100,
"filter": "update_ymdhi <= \"{$to_date}\" AND update_ymdhi > \"{$from_date}\""
{rdelim}{/capture}
{assign var=method_params value=$smarty.capture.method_params|json_decode:1}
{api_method
var=topics_list
model="Topics"
method="list"
version="1"
method_params=$method_params
}
{if $topics_list.errors|@count eq 0 && $topics_list.pageInfo.totalPageCnt > 0}
{section name=pager loop=$topics_list.pageInfo.totalPageCnt}
{assign var=request_params value=$empty_array}
{append var=request_params index=pageID value=$smarty.section.pager.iteration}
{api_method
var=topics_list
model="Topics"
method="list"
version="1"
method_params=$method_params
request_params=$request_params
}
{foreach from=$topics_list.list item=topics}
{assign var=row value=$empty_array}
{foreach from=$header_columns item=_v key=index}
{append var=row index=$index value=$topics.$index}
{/foreach}
{write_file path=$path value=$row is_append=1}
{write_file path=$path value="\n" is_append=1}
{/foreach}
{/section}
{/if}
Even within a batch process, memory overflow may occur when a large number of items are obtained at once. It is recommended to use the paging function.
Save the data
Finally, save the temporary file in an accessible directory.
Enter the following in the editor:
{put_file tmp_path=$path path="/files/user/topics_log/topics_"|cat:$to_date|cat:".csv"}
6. Save the Batch Process
After completing the description, click the [Add] button to save the process.
7. Check the File
After the batch process execution time, you can obtain the CSV file from the file manager.
