Situation
I'm currently in the first year of my apprenticeship, and my supervisor asked me to write a module for Icinga 2 Web (our monitoring tool) which makes it easier to delete old entries from our Influx database. The module displays all entries in a table and by clicking on a row and a button the entry will be removed from the database.
Since I'm somewhat confident with Python (4 months "experience") I wrote the script to get data from the database in Python and the rest in PHP, JavaScript and HTML (learned it for the task, so just 2 month experience). All scripts lies on the same Linux server inside the module folder.
The Python script, when executed, fetches the data out of InfluxDB, formats it to JSON and prints it, e.g.:
{'key': [{'key': value, 'key': value, ...}], 'key': [...], ...}
The JS part in my .phtml file gets the data by calling the Python script via
<script>
[...]
const dataJson = <?php echo shell_exec("python3 /[...]/get_db_data.py") ?>;
[...]
</script>
In my testing environment everything worked like a charm, but I only had 300 entries to process.
Problem
The database on our production server contains round about 200,000 entries. I fixed some bugs and managed to reduce the processing time of my Python script from 1 hour to ~17 second (I removed two sort functions...).
I noticed that the JS part failed to define the dataJSON. By pressing F12, I saw that the <?php [...] ?> is missing in the source code. Varying the number of entries fetched by the script it shows that as soon the number of entries surpasses 2,676 entries (= 53,299 characters) the error occurs.
[Script] dataJson = ;
[Console] Uncaught SyntaxError: expected expression, got ';'
With fewer entries everything works as intended. Am I surpassing some limits as maximum characters or something?
Things I tried
- Checked if the scripts works in general with all entries by running it from the terminal → It does.
- Using async/await-function as I thought the JavaScript would continue without waiting for the Python script to finish.
But it ends in the same error as described above.
<script>
[...]
async function getData() {
return await <?php echo shell_exec("python3 /[...]/get_db_data.py") ?>;
}
(async () => {
buildTable(await getData())
})()
[...]
</script>
- Tried to use node.js and child_process but it didn't work.
I didn't look further into this since I try to avoid installing new software to our production servers. In the end it told me that require is not defined (I failed to import it properly I guess).
My question
What am I doing wrong? Most likely I'm missing some crucial basics or using the different languages not properly. Since it is the last piece in this project and I will be moved to another department in a few days (in the three years of my apprenticeship, I only work for two to three months in each department of my company.) it is truly frustrating to get stuck at such error...
Thanks in advance :)