Creating daily backup data in CSV using Batch Process

Overview

Batch process is primarily used to automate time-consuming tasks or to periodically execute processes. In this exercse, we will implement a batch process that creates backup data on a daily basis

Before you start

In this tutorial, CSV will be saved under the following conditions:

• Batch process will run every day at 00:00.
• It will retrieve contents that belong to topics_group_id=5 and were updated the previous day.
• The content to be saved includes the content ID, title, body, extended item 1, and last updated date.
• A file will be created to store the retrieved content and saved in /files/user/topics_log/topics_yyyy-mm-dd.csv.

Creating Batch Process

Let's create a batch process.

1. Accessing the Batch Process list page

Click [Operations] -> [Batch process] on the menu to display the batch process list page.

2. Accessing the Batch Process editor page

Click [Add] in the upper right corner of the batch process list page.

The batch process editor page will be displayed.

3. Enter the Title and Identifier

First, enter the title and identifier as follows:

• Title: topics daily export
• Identifier: sample1_export_topics
• Batch: Every day, 00:00
• Member ID: Not specified

tip

Titles and identifiers must be unique and cannot have the same name with other batch processes.

4. Writing the daily backup process

Next, we will write the code for the daily backup process in the editor.

Output the header
First, output the header row of the CSV to a temporary file.
Enter the following in the editor:

caution

Replace the topics_group_id value=5 in the second line with the ID of the content structure in your site.

{assign var=empty_array value='[]'|json_decode:1}
{assign var=topics_group_id value=5}

{* Execute as admin *}
{login member_id=1}

{* Get topics group *}
{capture name=method_params}{ldelim}
"topics_group_id": [{$topics_group_id}],
"ext_config_flg": true
{rdelim}
{/capture}
{assign var=method_params value=$smarty.capture.method_params|json_decode:1}
{api_method
    var=topics_group
    model="TopicsGroup"
    method="details"
    version="1"
    method_params=$method_params
}

{* Create header line *}
{assign var=header_columns value=$empty_array}
{append var=header_columns index=topics_id    value="Topic ID"}
{append var=header_columns index=subject      value="Title"}
{append var=header_columns index=contents     value="Content"}
{* Additional field *}
{foreach from=$topics_group.ext_config item=config}
    {if $config.ext_col_nm eq 'ext_1'}
        {append var=header_columns index=$config.ext_col_nm value=$config.title}
    {/if}
{/foreach}
{append var=header_columns index=update_ymdhi value="Last update"}

{write_file var=path value=$header_columns}
{write_file path=$path value="\n" is_append=1}

Writing the process of retrieving content
Next, we will retrieve the content and write it immediately after the header row.
Enter the following in the editor:

{* Fetch contents updated one day before *}
{assign var=from_date value="-2 day"|strtotime|date_format:"%Y-%m-%d"}
{assign var=to_date value="-1 day"|strtotime|date_format:"%Y-%m-%d"}
{capture name=method_params}{ldelim}
    "topics_group_id": [{$topics_group_id}],
    "cnt": 100,
    "filter": "update_ymdhi <= \"{$to_date}\" AND update_ymdhi > \"{$from_date}\""
{rdelim}{/capture}
{assign var=method_params value=$smarty.capture.method_params|json_decode:1}
{api_method
    var=topics_list
    model="Topics"
    method="list"
    version="1"
    method_params=$method_params
}

{if $topics_list.errors|@count eq 0 && $topics_list.pageInfo.totalPageCnt > 0}
    {section name=pager loop=$topics_list.pageInfo.totalPageCnt}
        {assign var=request_params value=$empty_array}
        {append var=request_params index=pageID value=$smarty.section.pager.iteration}
        {api_method
            var=topics_list
            model="Topics"
            method="list"
            version="1"
            method_params=$method_params
            request_params=$request_params
        }
        {foreach from=$topics_list.list item=topics}
            {assign var=row value=$empty_array}
            {foreach from=$header_columns item=_v key=index}
                {append var=row index=$index value=$topics.$index}
            {/foreach}
            {write_file path=$path value=$row is_append=1}
            {write_file path=$path value="\n" is_append=1}
        {/foreach}
    {/section}
{/if}

tip

Even within a batch process, memory overflow may occur when a large number of data are retrieved at once. It is recommended to use the paging function.

Save the data
Finally, we need to save the temporary file in an accessible directory.
Enter the following in the editor:

{put_file tmp_path=$path path="/files/user/topics_log/topics_"|cat:$to_date|cat:".csv"}

6. Save the Batch Process

After completing the above configurations, click the [Add] button to save the batch process.

7. Check the file

After the batch process has been executed, you can obtain the CSV file from the file manager.

Related Documents

• Save daily data as CSV using batch processing
• Can the API respond in formats other than JSON?