AutoFS: Automated Feature Selection via Diversity-aware Interactive Reinforcement Learning Wei Fan∗, Kunpeng Liu∗, Hao Liuy, Pengyang Wang∗, Yong Gez, Yanjie Fu∗ ∗ Department of Computer Science, University of Central Florida, Orlando y Business Intelligence Lab, Baidu Research, Beijing z Eller College of Management, University of Arizona, Tucson ∗fweifan, kunpengliu, [email protected], [email protected], [email protected], ∗[email protected] Abstract—In this paper, we study the problem of balanc- Environment ing effectiveness and efficiency in automated feature selection. (Selected Feature Feature selection is to find the optimal feature subset from Subset) large-scale feature space, and is a fundamental intelligence for Actions machine learning and predictive analysis. After exploring many Advise (Select/ State s’ Reward r’ feature selection methods, we observe a computational dilemma: Deselect) 1) traditional feature selection methods (e.g., K-Best, decision Trainer tree based ranking, mRMR) are mostly efficient, but difficult to identify the best subset; 2) the emerging reinforced feature Agents in State s selection methods automatically navigate feature space to explore Reinforced Feature Selection the best subset, but are usually inefficient. Are automation Interactive Reinforced Feature Selection and efficiency always apart from each other? Can we bridge Fig. 1. Illustration of Interactive Reinforced Feature Selection (IRFS). the gap between effectiveness and efficiency under automation? Compared with basic reinforced feature selection, IRFS integrates the advice from the external teacher-like trainer. Motivated by such a computational dilemma, this study is to develop a novel feature space navigation method. To that process. Through preliminary exploration, we have observed end, we propose an Interactive Reinforced Feature Selection an interesting computational dilemma in feature selection: (IRFS) framework that guides agents by not just self-exploration experience, but also diverse external skilled trainers to accelerate (1) classic feature selection methods are mostly efficient, but learning for feature exploration. Specifically, we formulate the difficult to identify the best subset; 2) the emerging reinforced feature selection problem into an interactive reinforcement learn- feature selection methods automatically navigate feature space ing framework. In this framework, we first model two trainers to explore the best subset, but are usually inefficient. Are skilled at different searching strategies: (1) KBest based trainer; automation and efficiency always apart from each other? Can (2) Decision Tree based trainer. We then develop two strategies: (1) to identify assertive and hesitant agents to diversify agent we bridge the gap between effectiveness and efficiency under training, and (2) to enable the two trainers to take the teaching automation to strive for a balance? Motivated by the above role in different stages to fuse the experiences of the trainers dilemma, we focus on developing a new insight for addressing and diversify teaching process. Such a hybrid teaching strategy the effectiveness-efficiency balancing problem. In this paper, can help agents to learn broader knowledge, and thereafter be our key insight is to integrate both self-exploration experience more effective. Finally, we present extensive experiments on real- world datasets to demonstrate the improved performances of and external skilled trainers to simultaneously achieve both our method: more efficient than existing reinforced selection and effectiveness and efficiency in feature exploration. To this end, more effective than classic selection. two questions arise toward the goal. First, how can we introduce external skilled trainers to I. INTRODUCTION arXiv:2008.12001v3 [cs.LG] 16 Sep 2020 improve feature selection? Recently, Interactive Reinforcement In this paper, we aim to develop a new method to balance Learning (IRL) [7], [8], as an emerging technology, has shown the effectiveness and efficiency in automated feature selection. its superiority on accelerating the exploration of reinforcement Feature selection is to find the optimal feature subset from learning through learning from external knowledge. Such large feature space, which is a fundamental component for a interactive learning paradigm provides great potential for us wide range of data mining and machine learning tasks. to analogously introduce feature selection trainers to guide Classic feature selection methods include: filter methods agents to explore and learn more efficiently. To this end, (e.g., univariate feature selection [1]), wrapper methods (e.g., we reformulate feature selection problem into an interactive branch and bound algorithms [2]), and embedded methods reinforcement learning framework, which we call Interactive (e.g., LASSO [3]). Recently, the emerging reinforced feature Reinforced Feature Selection (IRFS), as illustrated in Figure 1. selection methods [4]–[6] formulate feature selection into a We propose a multi-agent reinforcement learning component reinforcement learning task, in order to automate the selection for feature selection, where an agent is a feature selector which issues select/deselect action toward its corresponding • Yanjie Fu is the contact author. feature; we integrate the interactive learning concept into multi-agent reinforcement learning, in order to bring in prior TABLE I knowledge from external trainers (e.g., K-Best, decision tree). NOTATIONS. These trainers can teach agents their selection experiences by Notations Definition j · j The length of a set recommending actions. The agents take recommended actions [x] The greatest integer less than or equal to x to improve the efficiency of their policy functions. x 1 − x N Number of features After introducing external trainers, how can we develop a fi The i-th feature new teaching strategy to enable external trainers to effectively agti The i-th agent πi The policy network of i-th agent guide agents? Intuitively, we directly consider a classical ai The actions of i-th agent feature selection method as a trainer (e.g., K-Best algorithm). ri The reward assigned to i-th agent However, this initial idea has a weakness: given a static feature set, single trainer always identifies similar feature Definition 2.2. Actions. Multiple agents corporately make subsets; that is, the trainer only provides mostly similar or decisions to select a feature subset. For a single agent, its the same advice to agents every time, which jeopardizes agent action space ai contains two actions, i.e., select and deselect. learning. Through preliminary exploration, we have identified Definition 2.3. State. The state s is defined as the represen- an interesting property of agent learning; that is, diversity tation of the environment, which is the selected feature subset matters. Our initial insight is to diversify the set of external in each step. trainers. So, we propose a KBest based trainer and a Decision Definition 2.4. Reward. The reward is to inspire the feature Tree based trainer, and thus both of they can teach agents to subspace exploration. We measure overall reward based on the learn more about feature space. To improve this initial insight, predictive accuracy in the downstream task, and equally assign we propose the second insight: to diversify the advice by the reward to the agents that select features at current step. selecting set of agents that will receive advice. Specifically, Definition 2.5. Trainer. In the apprenticeship of reinforce- we randomly select a group of agents each step, which we call ment learning, the actions of agents are immature, and thus it participated agents (features), to advise from trainers. In this is important to give some advice to the agents. we define the way, we add randomness to improve robustness. Moreover, source of the advice as trainer. we propose the third insight: different agents will receive Definition 2.6. Advice. Trainers give agents advice on their personalized and tailored advice. Specifically, we categorize actions. Multiple agents will follow the advice to take actions. the participated agents into: assertive agents (features) and Definition 2.7. Problem Statement. In this paper, we study hesitant agents (features). The assertive agents are more confi- the feature selection problem with interactive reinforcement dent to select their corresponding features, and thus don’t need learning. Formally, given a set of features F = ff1; f2; :::; fN g advice from trainers. The hesitant agents are less confident where N is the number of features, our aim is to find an 0 about their decisions, and thus need advice from trainers. optimal feature subset F ⊆ F which is most appropriate for Finally, we propose the fourth insight: to diversify the teaching the downstream task. In this work, considering the existence process. Specifically, we develop Hybrid Teaching strategy to of N features, we create N agents fagt1; agt2; :::; agtN g iteratively let various trainers take the teacher role at different correspondingly for feature ff1; f2; :::; fN g. All the agents use stages, to allow agents to learn a wide range of knowledge. their own policy networks fπ1; π2; :::; πN g to make decisions In summary, in this paper, we develop a diversity-aware in- to select or deselect their corresponding features, where the teractive reinforcement learning paradigm for feature selection. actions are denoted by fa1; a2; :::; aN g. For i 2 [1;N]; ai = 1 Our contributions are as follows: (1) We formulate