This doesn't sound right:
- The film was tried to be masked
Strange passive. This is a wrong passive of "try."
Imagine the comparable:
Can you make sense of it?
Now, this would have worked for you:
- The film was tried [out] [with an audience], masked as a fictional movie.
This is a good one:
but surely this is another meaning of "try."
Also
- The people who made the movie had tried to make it look like a fictional movie.
This is reasonable.
Now, perhaps you had "try out" in your mind, which can be used about movies:
A History of Broadcasting in the United States: The image ... - Page
166 Erik Barnouw - 1970
The film was tried out in California, and seemed so effective that Carroll Newton scheduled it for the final Sunday of the campaign,
immediately after the Ed Sullivan program on CBS-TV.