3

I have trained a simple neural net, to make predictions based on three inputs. For examples sake lets say I m trying to work out "how many days does it take to complete an accademic study", where the data could be:

f1 = institution
f2 = student age
f3 = study topic

and the label is the total number of days to complete.

If you imagine the following data structure:

f1, f2, f3, label

A,  D,  G,   v
B,  E,  H,   w
C,  F,  I,   x
A,  E,  H,   y
A,  D,  G,   z

Is there anyway that I can work out how long each feature adds to the total number of days to complete, so for example I want to be able to say that if you choose f1 = A that 4 days will be added to the total days to complete the study.

I can work out how much difference there is between each choice, eg choosing f1 = A might be 7 days quicker than if you chose f1 = B, but I do not know how to roughly ( maybe average ) identify how long each choice represents.

smci
  • 331
  • 2
  • 12
Laura Baker
  • 131
  • 2
  • But do you have to use an NN? rather than LR? and is your dataset big enough that NN will be preferable to LR? Personally I'd do LR, at least as a first step to directly see the coefficients for each level of each feature. There are also tree-based methods. – smci May 07 '18 at 23:06
  • 1
    Hi smci. I am actually looking at lr and also decision trees as well. I have checked out lime following you reccomendation and am quite excited generally by that package! Thanks – Laura Baker May 12 '18 at 07:31

1 Answers1

1

Yes you're talking about training a model, then doing sensitivity analysis to determine the numerical influence of each explanatory variable in isolation:

Some considerations/tips:

  • you said the three features are categorical. Presumably student_age is ordered categorical, and the other two are unordered. But you didn't say how many levels they each have (cardinality), or how many bins. You don't necessarily want to (say) bin student_age in bins of 1 year each. You might bin in say 2/3-year increments, or quantiles (see the R cut command), e.g. student_age: [0-17, 18-20, 21-23, 24-28, 29-34, 35+]. Similarly, if you have a long tail of smaller institutions with few exemplars, you might bin them together, or bin by country/ region/ size/ focus/ ranking.

  • what is the product of cardinality of each variable, vs the size of the dataset? Do you have 1 exemplar for each possible combination? >10? <0.1? If you don't have that much data, you might prefer to use Linear Regression rather than NN, or at least a shallower NN rather than deep. With Linear Regression you can allow quadratic interaction terms, nonlinear terms etc - again

  • (with Linear Regression you can directly see the influence just by directly reading the model coefficients e.g. Princeton: -1.4, Princeton.Age22_24: -3.6)

  • if you specifically want to use NN, see the lime package. Also see the many articles/tutorials on sensitivity analysis with NNs, and CrossValidated answers on sensitivity analysis.

  • deep NNs suffer from a lack of explainability; you can view the many individual layer weights but it's near-impossible to picture how that translates into effect on output.

smci
  • 331
  • 2
  • 12