0

A weird issue started happening in higher environments while no problems have been logged in the development and testing environments with identical databases. The notable difference is that the two higher environemnts include a load balancer.

Existing Design: An angular 10.? is hosted as a custom element inside of a traditional .net aspx page. The angular app makes various async ajax calls in promises to web methods in the host aspx page. In turn, the aspx page translates the request and calls a matching async .NET Core web api 2.0 endpoint. The entire call chain uses an async pattern, except for the web method in the aspx page. Well, that call calls the Result of a Task resulting in synchronous execution.

Problem: This has been tested extensively on large and small datasets in development and testing environments. When the code was pushed up to higher environments, with load balancing, a weird event started sporadically occurring. Two of the 20'ish api calls sporadically can get "stuck" in a "pending" http request state as seen through wire shark and F12 dev tools, but only two or the 20 or more. The fun part is that after ~4-5 minutes the calls return. SQL trace indicates all SP calls have completed without issue. The dataset is large but the payload is not an issue because when the calls work on the same shape and size of data the response takes ~10 seconds. So I am at a loss of why a wayward http call using the PerformGetAsync() call below that usually takes ~15 seconds is returning in 5 minutes.

My initial thought was parameter sniffing or a poor query plan was cached, however, that was ruled out via a session of sql profiler. Then thought turned to the load balancer and how it could delay an http response and keep it in pending state for 5 minutes (I am still researching that). While researching the async call chain, the lack of await on GetAsync() and calling the .Result on the task stood out, however, it seems if that is removed the angular app's calls never get the response or return prior to the call finishing.

I started reading up on some tech documents stating that when an IIS instance gets busy enough it can spin threads up on a custom thread pool and certain codes can cause a deadlock by calling .Result on the main thread (the same way a winforms app UI thread can lock), however, if that was the case then why would the response return after ~5 minutes. So, I am still looking into a load balancer being an issue here.

Sorry for the wall of text, the point was to be as detailed as possible.

Angular call:

  getSomething(request: GetSomethingRequest): Observable<ApiResponse> {
    
    const url = environment.appUrls.something + 'GetSomething';

    return this.http
      .post(url, `{ "request": ${JSON.stringify(request)} }`, this.httpOptions)
      .pipe(
        map((x) => this.toObject(x) as any),
        catchError(this.handleError('getSomething'))
      );
  }

aspx Web Method Called by hosted angular app:

<WebMethod>
<ScriptMethod(ResponseFormat:=ResponseFormat.Json)>
Public Shared Async Function GetSomething(request As GetSomethingRequest) As Task(Of GetResponse)
    Dim urlString As String = "/Something/" + request.SomethingID.ToString()
    Return Await PerformGetAsync(urlString)
End Function

Shared Code (called from .aspx web method above):

private async static Task<GetResponse> PerformGetAsync(string url)
{
    try
    {
        EnableGZip();

        var client = new HttpClient();
        client.DefaultRequestHeaders.Add("HeaderValue1", "ABC");
        client.DefaultRequestHeaders.Add("HeaderValue2", "123");

        var response = client.GetAsync(url).Result;

        var getResponse = new GetResponse({
            data = await response.Content.ReadAsStringAsync(),
            success = response.StatusCode == HttpStatusCode.OK,
            statusCode = (int)response.StatusCode
        });

        return getResponse;
    }
    catch(Exception ex)
    {
        Logger.LogException(ex);

        return new GetResponse({
            data = string.Empty,
            success = false,
            statusCode = (int)HttpStatusCode.InternalServerError
        });
    }
}

In .NET core web api

[HttpGet("Something/{SomethingId}")]
public async Task<ActionResult<GetResponse>> GetSomethingAsync([FromRoute] GetRequest request)

NOTE : The code is compile-able and runs very fast. I have modified it slightly in this post to remove some of the details

Ross Bush
  • 13,809
  • 2
  • 33
  • 52

0 Answers0