Sitelet https://github.com/Shuffle/python-apps/issues/428
Skip to content

Timeout error in HTTP app #428

Open
Open
Timeout error in HTTP app#428

Description

@Kagetana
Kagetana
opened on Jun 17, 2025

HTTP app version 1.4.0

When I'm trying to GET alerts from DFIR IRIS with http app or "custom action" in IRIS app i get an error/
IRIS endpoint: /alerts/filter
this endpoint by default return 10 alerts and total number of pages, and in this case everything works fine,
But when i'm adding query params like this per_page=100&creation_start_date=$exec.start_date&creation_end_date=$exec.end_date i get an error because of timeout (more than 10 alerts takes time to process by IRIS and return response)

shuffle_action_logs (sanitized):

2025-06-17 09:38:35,304 - HTTP - INFO - [INFO][a1269c43-3993-45eb-bba7-24b04cdf2fcd] Action name: GET, Params: 9, Has Body: False
2025-06-17 09:38:35,318 - HTTP - INFO - [INFO][a1269c43-3993-45eb-bba7-24b04cdf2fcd] Running function 'GET' with timeout 30
2025-06-17 09:38:35,323 - HTTP - INFO - ['Accept: application/json', 'Authorization: Bearer correct_IRIS_API_token']
2025-06-17 09:38:40,365 - HTTP - INFO - [ERROR] Something is wrong with the input for this function. Are lists and JSON data handled parsed properly (1)? err: HTTPSConnectionPool(host='myirisdomain.org', port=443): Read timed out. (read timeout=5)
2025-06-17 09:38:40,365 - HTTP - INFO - [DEBUG][a1269c43-3993-45eb-bba7-24b04cdf2fcd] Sending result to https://myshuffledomain.org/api/v1/streams

My idea to fetch all alerts for some period of time, like per_page=10000 and process them all in my workflow, so i wanna set timeout like 10-30mins for this action
if i read code of http app properly timeout=5 is hardcoded, my suggestion to add possibility to change this value

Activity

  1. Kagetana commented on Jun 17, 2025

    @Kagetana
    Kagetana
    on Jun 17, 2025
    Author

    Correction:
    I guess I'm blind, it's already there
    But it doesn't help me with value like 600
    {"success":false,"reason":"POST error: Failed connecting to https://shuffle1.hosterby.com/api/v1/streams over 10 retries to the backend"}

  2. Kagetana commented on Jun 17, 2025

    @Kagetana
    Kagetana
    on Jun 17, 2025
    Author

    Execution with fetching almost 2k alerts in console from shuffle host handled approximately in 20 sec
    Curl/http/IRIS apps got stuck, for now have no idea how to bypass this problem

  3. frikky commented on Jun 17, 2025

    @frikky
    frikky
    on Jun 17, 2025
    Member

    Correction: I guess I'm blind, it's already there But it doesn't help me with value like 600 {"success":false,"reason":"POST error: Failed connecting to https://shuffle1.hosterby.com/api/v1/streams over 10 retries to the backend"}

    This sounds like a backend problem on the Shuffle database side, which usually stems from too large amounts of data being returned.

    Could you check the shuffle-backend container at the same time as this app error showed up? docker logs shuffle-backend

  4. Kagetana commented on Jun 18, 2025

    @Kagetana
    Kagetana
    on Jun 18, 2025
    Author

    this is backend logs:

    2025/06/18 06:42:17 [INFO] Inside execute workflow for ID 5409033f-0930-4c82-9f4c-0cb8bfd3899d
2025/06/18 06:42:17 [AUDIT] Letting user soc@domain.org execute 5409033f-0930-4c82-9f4c-0cb8bfd3899d because they're admin of the same org
2025/06/18 06:42:17 [AUDIT] Starting execution of workflow '5409033f-0930-4c82-9f4c-0cb8bfd3899d' by user 'soc@domain.org' (999195d8-745c-47a7-a000-34022375979a). If this is empty, it's most likely a subflow!
2025/06/18 06:42:17 [WARNING] Failed hashing in encrypt: No encryption modifier set. Define SHUFFLE_ENCRYPTION_MODIFIER and NEVER change it to start encrypting auth.
2025/06/18 06:42:17 [INFO][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Execution: should execute onprem with execution environment "hoster". Workflow: 5409033f-0930-4c82-9f4c-0cb8bfd3899d
2025/06/18 06:42:25 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Running workflow execution update with result from Shuffle_Tools_2 (4644e114-fc8e-49d4-99cd-2a6cbaeebad2) of status SUCCESS
2025/06/18 06:42:43 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Running workflow execution update with result from http_1 (08b1fa54-ad9b-4227-9b91-7200c728b274) of status FAILURE
2025/06/18 06:42:43 [DEBUG] Result is FAILURE for http_1 (08b1fa54-ad9b-4227-9b91-7200c728b274). Making notification.
2025/06/18 06:42:43 [INFO] Reopening notification with title "Error in Workflow KPI test" for users in org f5c5b32b-d445-49c5-8fbb-d6d182307f5a
2025/06/18 06:42:43 [WARNING] Actionresult is FAILURE for node 08b1fa54-ad9b-4227-9b91-7200c728b274 in 294e965c-60d2-4284-95d4-dfa7dae1a7b5. Continuing anyway because of workflow configuration.
2025/06/18 06:42:43 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Checking result curl_IRIS_API_test_alert_start_date (115aa82d-6111-47f3-b56c-c4da24824c79) with status SKIPPED
2025/06/18 06:42:43 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Checking result curl_IRIS_API_test_owner_id (152d166a-d40e-4986-920d-9a0b4fea24e0) with status SKIPPED
2025/06/18 06:42:43 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Checking result Shuffle_Tools_4 (67a940fd-ad90-4a4b-b7e3-684c487385f6) with status SKIPPED
2025/06/18 06:42:43 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Checking result IRIS_v2_1 (0f19bb95-60cd-40c2-a54b-673ccca20b22) with status SKIPPED
2025/06/18 06:42:43 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Checking result Shuffle_Tools_2 (4644e114-fc8e-49d4-99cd-2a6cbaeebad2) with status SUCCESS
2025/06/18 06:42:43 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Setting execution to aborted because of result GET (08b1fa54-ad9b-4227-9b91-7200c728b274) with status 'FAILURE'. Should update execution parent if it exists (not implemented).
2025/06/18 06:42:43 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Setting execution to aborted because of result GET (08b1fa54-ad9b-4227-9b91-7200c728b274) with status 'FAILURE'. Should update execution parent if it exists (not implemented).
2025/06/18 06:42:44 [AUDIT] User soc@domain.org (999195d8-745c-47a7-a000-34022375979a) is accessing workflow 'KPI test' (5409033f-0930-4c82-9f4c-0cb8bfd3899d) executions as admin (get executions)
2025/06/18 06:42:44 [INFO][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Set workflowexecution based on new worker (>0.8.53) for workflow 5409033f-0930-4c82-9f4c-0cb8bfd3899d. Actions: 7, Triggers: 0, Results: 6, Status: ABORTED
2025/06/18 06:42:48 [WARNING] Continuing workflow even though it's aborted (ExitOnError config)
2025/06/18 06:42:48 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Running workflow execution update with result from Shuffle_Tools_5 (341ab06d-e584-4bb6-9e84-1df09357a8b7) of status FAILURE
2025/06/18 06:42:48 [DEBUG] Result is FAILURE for Shuffle_Tools_5 (341ab06d-e584-4bb6-9e84-1df09357a8b7). Making notification.
2025/06/18 06:42:48 [INFO] Reopening notification with title "Error in Workflow KPI test" for users in org f5c5b32b-d445-49c5-8fbb-d6d182307f5a
2025/06/18 06:42:48 [WARNING] Actionresult is FAILURE for node 341ab06d-e584-4bb6-9e84-1df09357a8b7 in 294e965c-60d2-4284-95d4-dfa7dae1a7b5. Continuing anyway because of workflow configuration.
2025/06/18 06:42:48 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Checking result curl_IRIS_API_test_alert_start_date (115aa82d-6111-47f3-b56c-c4da24824c79) with status SKIPPED
2025/06/18 06:42:48 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Checking result curl_IRIS_API_test_owner_id (152d166a-d40e-4986-920d-9a0b4fea24e0) with status SKIPPED
2025/06/18 06:42:48 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Checking result Shuffle_Tools_4 (67a940fd-ad90-4a4b-b7e3-684c487385f6) with status SKIPPED
2025/06/18 06:42:48 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Checking result IRIS_v2_1 (0f19bb95-60cd-40c2-a54b-673ccca20b22) with status SKIPPED
2025/06/18 06:42:48 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Checking result Shuffle_Tools_2 (4644e114-fc8e-49d4-99cd-2a6cbaeebad2) with status SUCCESS
2025/06/18 06:42:48 [DEBUG][294e965c-60d2-4284-95d4-dfa7dae1a7b5] Checking result http_1 (08b1fa54-ad9b-4227-9b91-7200c728b274) with status FAILURE

    So Shuffle can't process large amount of data? Or problem not that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions